Closed fcbond closed 4 years ago
The |
character is specifically excluded from TDL identifiers. See this part of the TdlRfc wiki.
The reason is because #|
is the sequence to start a block comment, but if |
is allowed as an identifier, it is ambiguous with a coindexation (e.g., [ FOO #|, BAR.BAZ #| ]
). See this message on the developer's list. This is therefore a problem with the ERG and not PyDelphin.
I don't think ACE or the old LKB has been revised to match the new specification, but LKB-FOS has. Can you try it with LKB-FOS to confirm?
@danflick @oepen I don't have access to the latest ERG so I cannot test this, but Francis uncovered a problem with an identifier using an invalid character. I found that this is actually in the current trunk already, but I wasn't testing PyDelphin on files in subdirectories like educ/
. Until I can see the ERG 2020 files, you can run something like this to look for other TDL errors:
find . \
-name \*.tdl \
! -name \*config\*.tdl \
-exec echo {} \; \
-exec python -c 'import sys, delphin.tdl; list(delphin.tdl.iterparse(sys.argv[1]))' {} \;
(I ignore *config*.tdl
because ACE configs are not really TDL)
Running this on the current trunk branch uncovered two PyDelphin bugs related to TDL parsing (#293 and #294), so be aware of those as you test.
It would actually be helpful if I could also parse the config.tdl for the ltsb.
It mainly parses, the only problem is things like this:
preprocessor-modules := ../rpp/xml.rpp ../rpp/ascii.rpp ../rpp/quotes.rpp.
I fear these are not legal TDL --- can they be made so in some way?
On Fri, Jun 5, 2020 at 9:47 AM Michael Wayne Goodman < notifications@github.com> wrote:
@danflick https://github.com/danflick @oepen https://github.com/oepen I don't have access to the latest ERG so I cannot test this, but Francis uncovered a problem with an identifier using an invalid character. I found that this is actually in the current trunk already, but I wasn't testing PyDelphin on files in subdirectories like educ/. Until I can see the ERG 2020 files, you can run something like this to look for other TDL errors:
find . \ -name *.tdl \ ! -name *config*.tdl \ -exec echo {} \; \ -exec python -c 'import sys, delphin.tdl; list(delphin.tdl.iterparse(sys.argv[1]))' {} \;
(I ignore config.tdl because ACE configs are not really TDL)
Running this on the current trunk branch uncovered two PyDelphin bugs related to TDL parsing (#293 https://github.com/delph-in/pydelphin/issues/293 and #294 https://github.com/delph-in/pydelphin/issues/294), so be aware of those as you test.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/delph-in/pydelphin/issues/292#issuecomment-639209185, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIPZRRLJFOYDOLEKLVSJMTRVBFBJANCNFSM4NSTZHDA .
-- Francis Bond http://www3.ntu.edu.sg/home/fcbond/ Division of Linguistics and Multilingual Studies Nanyang Technological University
@fcbond I have some more radical thoughts on this which I may bring up at the virtual summit.
Less radically, the file could include some way to indicate it's not regular TDL:
.tdl
extension:begin :config ace.
...
:end :config.
The last option introduces a change to TDL, but a backward compatible one I think.
Hi Mike,
Yes, that illegal character should have been in a commented-out definition, as it's part of a little experiment to see if I could have a single lexical mal-entry trigger two distinct error messages. I'll figure out a TDL-legal way to do that, and in the meantime, I've commented out that problematic definition in mo', soon to be
trunk'. I'll run your script, thanks, and see what else needs attending to.
Dan
From: Michael Wayne Goodman notifications@github.com Sent: Thursday, June 4, 2020 6:47 PM To: delph-in/pydelphin pydelphin@noreply.github.com Cc: Dan Flickinger danf@stanford.edu; Mention mention@noreply.github.com Subject: Re: [delph-in/pydelphin] Pydelphin cannot parse used_to_prp_v_rbst|use_to_aux_sp_rbst := v_p-vp_prp_le & (#292)
@danflickhttps://github.com/danflick @oepenhttps://github.com/oepen I don't have access to the latest ERG so I cannot test this, but Francis uncovered a problem with an identifier using an invalid character. I found that this is actually in the current trunk already, but I wasn't testing PyDelphin on files in subdirectories like educ/. Until I can see the ERG 2020 files, you can run something like this to look for other TDL errors:
find . \ -name *.tdl \ ! -name *config*.tdl \ -exec echo {} \; \ -exec python -c 'import sys, delphin.tdl; list(delphin.tdl.iterparse(sys.argv[1]))' {} \;
(I ignore config.tdl because ACE configs are not really TDL)
Running this on the current trunk branch uncovered two PyDelphin bugs related to TDL parsing (#293https://github.com/delph-in/pydelphin/issues/293 and #294https://github.com/delph-in/pydelphin/issues/294), so be aware of those as you test.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/delph-in/pydelphin/issues/292#issuecomment-639209185, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AG5PC4WAZQL3RZU4K7MFHA3RVBFBJANCNFSM4NSTZHDA.
Pydelphin cannot parse this (from the latest ERG-makeover), but ace can
If I replace '|' with '___' it parses fine.