Closed lmorgadodacosta closed 4 years ago
One thing that might be an issue is the escaped tabs (\t
) as TSDB only has 3 escapes:
\n
- newline\s
- @ character\\
- backslashSo that escaped tab should be \\t
or just an actual tab character, depending on what was in the source.
I'm also assuming that all the newlines between these records were inserted by you (e.g., by reading and printing each line in Python without doing line.rstrip("\n")
) and aren't actual blank lines in the file.
Does any of that resonate with what you see?
So, on emacs each item appears in a new line. But when pasted here they did collapse, so I added an extra line... This is the true copy-paste:
20021000050@1@1@-1@lmc@30-06-2020 03:31:42@30-06-2020 03:41:28@
20021000100@1@0@-1@lmc@30-06-2020 03:42:00@30-06-2020 03:49:16@
20021000120@1@0@-1@lmc@30-06-2020 03:49:28@30-06-2020 03:53:06@0 - 'view on' - 'on' is redundant; 'login in' should be 'logging in'. hdn_bnp_c \shdn_optcmp_c \sw_hasinitcap_dlr \sn_pl_olr \sn_pp_c-of_le\t=\t0 to 1\t[x]\nsp-hd_n_c\t=\t7 to 10\t[x]\nsp-hd_n_c\t=\t11 to 15\t[x]\naj-hdn_norm_c\t=\t13 to 15\t[x]\nv_j-nb-prp-tr_dlr \sv_prp_olr \sv_np*_le\t=\t13 to 14\t[x]\nnp-np_crd-t_c\t=\t7 to 15\t[x]\nthird_sg_fin_v_rbst \sv_-_le\t=\t16 to 17\t[x]\nn-hdn_cpd_c\t=\t24 to 27\t[x]\nn-hdn_cpd_c\t=\t20 to 22\t[x]\nd_-_poss-their_le\t=\t23 to 24\t[x]\nn_sg_ilr \sn_-_c-gr_le\t=\t25 to 26\t[x]\np_np_i_le\t=\t22 to 23\t[x]\nhd_xcmp_c \sv_n3s-bse_ilr \sv_np_le\t=\t5 to 6\t[x]\ncm_vp_to_le\t=\t4 to 5\t[x]\nv_psp_olr \sv_np_le\t=\t2 to 3\t[x]\nhd_xcmp_c \sp_np_i-reg_le\t=\t17 to 18\t[x]\np_np_i-nm-no-tm_le\t=\t18 to 19\t[x]\nn-hdn_cpd-pl_c\t=\t8 to 10\t[x]\n system rejects
20021000020@1@1@-1@lmc@30-06-2020 03:53:57@30-06-2020 04:03:34@not quite sure how to parse. 'for consumers... particular restaurant'.
20021000010@1@1@-1@lmc@30-06-2020 05:35:47@30-06-2020 05:39:34@
In addition to this weirdness, on some specific comments, I can also see that a new line ("\n") was attempted by the system between the "real comment" and the rubbish that comes with it (e.g. "adjective. \n" below)
20018000170@1@1@-1@lmc@30-06-2020 16:48:58@30-06-2020 16:49:37@
20015000100@1@0@-1@lmc@01-07-2020 01:15:39@01-07-2020 01:19:07@system rejected - doesn't allow 'serial' to be adjective. \nd_-_sg-nmd_le\t=\t3 to 4\t[x]\ndet_prt-nocmp_dlr \sw_hasinitcap_dlr \sd_-_prt-sg_le\t=\t0 to 1\t[x]\nhd_xcmp_c \sv_3s-fin_olr \sv_np_poss_le\t=\t2 to 3\t[x]\nn_sg_ilr \sn_-_c-nocnh_le\t=\t6 to 7\t[x]\naj_-_i_le\t=\t5 to 6\t[x]\n
20016000080@1@1@-1@lmc@01-07-2020 01:19:30@01-07-2020 01:20:29@
It feels more and more clear that this is an FFTB problem. I've asked the treebanker for details on the browser/machine they are using.
(edit: I changed these to use triple grave characters (```
) so newlines appear as they should)
The \n
escapes are not an issue from the TSDB side, although I wonder why the "rubbish" is being included at all (a question for the annotation setup).
I think you're correct that these are an issue with FFTB (or rather, libtsdb). If you have a suggestion for a more helpful error message I can consider that for PyDelphin. Perhaps it is possible to print out what exactly was the unexpected escaped character, for instance.
Woodley has accepted a patch for FFTB so that it does not escape \t
or \r
. So I'm closing this as it is not an issue with PyDelphin.
While trying to process the output of FFTB, I noticed that the "tree" file produced things that pydelphin's itsdb doesn't like. I have a hunch that this is not really pydelphin's problem, but I wanted to double check. You complain about not properly escaped characters. This seems FFTB might be writing some important information as comments. Although this only happened with one of our 5 treebankers.
Example of FFTB output of file 'tree':