ELind77
commented
9 years ago Re-read the blog post saw that you are using sbcl. Will try and set that up to sets out the project.
Closed ELind77 closed 7 years ago
ELind77
commented
9 years ago Re-read the blog post saw that you are using sbcl. Will try and set that up to sets out the project.
vseloved
commented
9 years ago Hi, thanks for the notice!
I believe, this would be a simpler solution:
(defparameter +white-chars+
(list #\Space #\Tab #\Newline #\Return #\Linefeed
(code-char 160)) ; #\No-Break_Space
"Chars considered WHITESPACE.")What do you think?
ELind77
commented
9 years ago That looks good to me!
Do you have any thoughts on the multi-byte chars later in that file?
vseloved
commented
9 years ago Yes, thanks for the remainder. I think, the chars issue will be handled the best with the conditional compilation like you've proposed:
#-lispworks '(#\" #\‘ #\’ #\« #\» #\“ #\” #\')
#+lispworks ...If you could provide a filling for the '...' for LispWorks as a pull request I'd be grateful, as I don't have access to that platform now.
ELind77
commented
9 years ago So, (code-char 160) worked fine. But something weird is going on with the conditional compilation for the other characters. I changed the parameters to look like this:
(defparameter +quote-chars+
#-lispworks '(#\" #\‘ #\’ #\« #\» #\“ #\” #\')
#+lispworks (mapcar #'code-char '(34 8216 8217 171 187 8220 8221 39))
"Chars considered legitimate quotation marks.")And when the #-lispworks line is commented out everything loads fine. But when I try and compile/load with the code as above, it hits the same error it did before and I have no idea why. I have no problem evaluating it in the listener but when I try to compile and load the file it freaks out. I typically just use emacs integration with LW to develop but I tried running it in the LW IDE itself to see what would happen and it couldn't even display the characters in the editor pane. Maybe LW can't handle non-ASCII-encoded source files? I'll try and look around to see if I can find anything for that.
But since I was able to compile the chars.lisp file I went ahead and fixed the next compiler error which was in the tokenize function. Lines 255 and 266 of that file also have some UTF-8 characters (the paragraph symbol and I put in conditional compilation for them and that actually worked. Which only adds to the mystery of the chars file.
So the tokenize function looks like this:
(defmethod tokenize ((tokenizer baseline-sentence-tokenizer) string)
(mv-bind (words word-spans)
(tokenize (make 'regex-word-tokenizer :regex "[^\\s]+")
(substitute
#-lispworks #\¶
#+lispworks (code-char 182)
#\Newline string))
(let ((beg 0)
sentences spans)
(loop :for ws :on words :and ss :on word-spans :do
(let ((word (first ws))
(span (first ss)))
(when (or (null (rest ws))
(and (member (char word (1- (length word)))
#-lispworks '(#\. #\? #\! #\¶)
#+lispworks '(#\. #\? #\! (code-char 182))
)
(not (member word +abbrevs-with-dot+
:test #'string-equal))
(and-it (second ws)
(upper-case-p (char it 0)))))
(push (sub string beg (rt span)) sentences)
(push (pair beg (rt span)) spans)
(setf beg (lt (second ss))))))
(values (reverse sentences)
(reverse spans)))))And then I hit one last error which completely threw me. The COUT-NGRAM-FREQS function in ngrams.lisp had an error that made no sense to me: ../core/ngrams.lisp:250:8: error: The variable #:|table3296| is unbound
And it was pointing to the :+ in the body of the LOOP. I tried a bunch of stuff but I really have no idea what the problem is. The weird part is that if I replace the #` function with a lambda it works fine in the listener but still fails when compiling the file.
After failing to fix that I tried to load up the files with sbcl 1.2.4 and that didn't work either :< (I have some kind of compatibility issue with 1.0.58 on my desktop). I have a backtrace for that but I wasn't sure if I should just copy-paste it here as this post is already getting quite long. Let me know what you want me to do with that.
-- Eric
vseloved
commented
9 years ago Hi Eric, thanks for the detailed description! It made me download and install LispWorks Personal Edition to figure out the issues you face there, and I can reproduce everything. Still, I also don't understand the reason for the unbound variable error. I'll try to fiddle with it.
Please, post the SBCL stacktrace as well.
ELind77
commented
9 years ago Here is the backtrace from the failed sbcl load-compile (I tried version 1.2.3 and 1.2.4, I can't get 1.0.58, which works, to run on my desktop because of some conflict):
The value #
Restarts: 0: [RETRY] Retry compiling #<CL-SOURCE-FILE "rutils" "core" "readtable">. 1: [ACCEPT] Continue, treating compiling #<CL-SOURCE-FILE "rutils" "core" "readtable"> as having been successful. 2: [RETRY] Retry ASDF operation. 3: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the configuration. 4: [ABORT] Give up on "rutils" 5: [RETRY] Retry SLIME REPL evaluation request. --more--
Backtrace:
0: (SET-MACRO-CHARACTER #( #
vseloved
commented
9 years ago The problem seems to be in readtable definition via named-readtables.
Can you evaluate the following form before loading cl-nlp: (sb-ext:restrict-compiler-policy 'debug 3), and provide an updated stacktrace? (It should switch on a more detailed output for the stack trace as currently the names of functions are represented simply by # signs).
I've found a couple of compatibility issues with the chars.lisp file in src/utils/ and LispWorks
the first was in the +WHITE-CHARS+ param, LispWorks uses #\NO-BREAK-SPACE so I did:
I expect there may be a better way to do this that fits with your project coding standards but I leave that integration to you. Example used for fix: link to CLSQL project
Once I put that in the compile got further into the file and I found a character encoding issue. Some of the quotation characters are multi-byte characters that LispWorks can't read properly. Emacs appears to have no problem displaying them, but when opened in Lispworks it doesn't display them properly and the compiler can't read the characters. LispWorks uses UTF-16 internally, and if there are char-codes for the characters you are using that are the same across UTF-8/16 that might work. There may also be a more elegant solution but I don't know enough about how LispWorks is treating the characters to figure anything else out.
I may just switch over to sbcl to test out this project. What lisp implementation are you developing in?
--Eric