Runtime error parsing token

marcoheisig / cl4py

Common Lisp for Python

MIT License

94 stars 14 forks source link

Runtime error parsing token #24

Closed dmenager closed 8 months ago

dmenager commented 9 months ago

I'm writing a Python program that calls a function in a loop that uses cl4py to talk to Lisp (SBCL). The Python code passes in a dotted list with the car being a list of Lisp structures, while the cdr of the dotted list is an integer-key dictionary. The Python program runs for several iterations, then fails with a: "RuntimeError: Failed to parse token "0:" "

Traceback (most recent call last):
  File "/home/david/Code/agile_manager_analytics.py", line 58, in <module>
    bn = row_to_hems_program(row, hems)
  File "/home/david/Code/agile_manager_analytics.py", line 25, in row_to_hems_program
    return hems.compile_program_from_file(fp.name)
  File "/home/david/.local/lib/python3.10/site-packages/cl4py/data.py", line 282, in __call__
    return self.lisp.eval(List(Symbol('FUNCALL', 'CL'), Quote(self), *restAndKeys))
  File "/home/david/.local/lib/python3.10/site-packages/cl4py/lisp.py", line 79, in eval
    pkg = self.readtable.read(self.stdout)
  File "/home/david/.local/lib/python3.10/site-packages/cl4py/reader.py", line 104, in read
    return self.read_aux(stream)
  File "/home/david/.local/lib/python3.10/site-packages/cl4py/reader.py", line 169, in read_aux
    return self.parse(''.join(token))
  File "/home/david/.local/lib/python3.10/site-packages/cl4py/reader.py", line 213, in parse
    raise RuntimeError('Failed to parse token "' + token + '".')
RuntimeError: Failed to parse token "0:".

Please see the attached small dataset to reproduce the error.

AgileCaseBase_No_Missing_KDMAs_reduced_ID_Merged.xlsx

The error may be reproduced with the following code:

import pandas as pd
import numpy as np
import cl4py
import tempfile
from cl4py import Symbol
from cl4py import List as lst

def row_to_hems_program (row, hems):
    with tempfile.NamedTemporaryFile() as fp:
        i = 1
        for index, value in row.items():
            if index == "User Strategy Index":
                fp.write(bytes("c0 = (percept-node {0} :value {1})\n".format(index.replace(" ", "_"), "\"" + str(value) + "\""), 'utf-8'))
            else:
                fp.write(bytes("c{0} = (percept-node {1} :value {2})\n".format(i, index.replace(" ", "_"), "\"" + str(value) + "\""), 'utf-8'))
                i += 1
        for j in range(i):
            if j > 0:
                fp.write(bytes("c0 -> c{0}\n".format(j), 'utf-8'))
        fp.seek(0)
        return hems.compile_program_from_file(fp.name)

# Get a handle to the lisp subprocess with quicklisp loaded.
lisp = cl4py.Lisp(quicklisp=True, backtrace=True)

# Start quicklisp and import HEMS package
lisp.find_package('QL').quickload('HEMS')

# Load hems and retain reference.
hems = lisp.find_package("HEMS")

# Load agile manager dataset.
df = pd.read_excel("AgileCaseBase_No_Missing_KDMAs_reduced_ID_Merged.xlsx").drop(['Game Level', 'Player Score', 'Player Score Loss (Low Quality)', 'Player Score Loss (Tardiness)', 'AI Score', 'AI Score Loss (Low Quality)', 'AI Score Loss (Tardiness)', 'ID', 'Country', 'Age'], axis=1)

for idx, row in df.iterrows():
    if idx >= -1:
        bn = row_to_hems_program(row, hems)
        print()
        print(idx)
        hems.push_to_ep_buffer(state=bn, insertp=True)

The code depends on a Lisp library I've written, which you can find on Github here. So, please clone that repo to your quickilsp local-projects directory before running the Python script.

Interestingly, I have a version of this code written completely in Lisp and it works fine, so I believe the error is occurring in the parser.

marcoheisig commented 9 months ago

I managed to reproduce your problem. I put your code snippet in a file named bug.py, and ran strace -e read -s 50000 python3 bug.py 2> log.txt to trace how the Python process communicates with its SBCL subprocess. Here is the message that breaks cl4py's parser:

   0: fp=0x7f9b87fad238 pc=0x52e2f2cf SB-IMPL::HASH-TABLE-NEW-VECTORS
   1: fp=0x7f9b87fad2a0 pc=0x52e2fa76 SB-IMPL::GROW-HASH-TABLE
   2: fp=0x7f9b87fad308 pc=0x52ba0cb9 (FLET SB-IMPL::INSERT-AT :IN \"SYS:SRC;CODE;TARGET-HASH-TABLE.LISP\")
   3: fp=0x7f9b87fad370 pc=0x52ba01d1 SB-IMPL::PUTHASH/EQL
   4: fp=0x7f9b87fad398 pc=0x52a1667c SB-KERNEL::%PUTHASH
   5: fp=0x7f9b87fad458 pc=0x535de3ef HEMS::UPDATE-CPD-RULES
   6: fp=0x7f9b87fad520 pc=0x535f0069 HEMS::FACTOR-FILTER
   7: fp=0x7f9b87fad618 pc=0x5362137a HEMS::COMBINE-STATES
   8: fp=0x7f9b87fad6e8 pc=0x53622901 HEMS::COMBINE
   9: fp=0x7f9b87fad7a0 pc=0x5362474a HEMS::EP-MERGE
  10: fp=0x7f9b87fad8b0 pc=0x53625b5f HEMS::INSERT-EPISODE
  11: fp=0x7f9b87fad9c0 pc=0x53625f78 HEMS::INSERT-EPISODE
  12: fp=0x7f9b87fadad0 pc=0x53625f78 HEMS::INSERT-EPISODE
  13: fp=0x7f9b87fadbe0 pc=0x53625f78 HEMS::INSERT-EPISODE
  14: fp=0x7f9b87fadcf0 pc=0x53625f78 HEMS::INSERT-EPISODE
  15: fp=0x7f9b87fade00 pc=0x53625f78 HEMS::INSERT-EPISODE
  16: fp=0x7f9b87fadf10 pc=0x53625f78 HEMS::INSERT-EPISODE
  17: fp=0x7f9b87fae020 pc=0x53625f78 HEMS::INSERT-EPISODE
  18: fp=0x7f9b87fae130 pc=0x53625f78 HEMS::INSERT-EPISODE
  19: fp=0x7f9b87fae240 pc=0x53625f78 HEMS::INSERT-EPISODE
  20: fp=0x7f9b87fae350 pc=0x53625f78 HEMS::INSERT-EPISODE
  21: fp=0x7f9b87fae460 pc=0x53625f78 HEMS::INSERT-EPISODE
  22: fp=0x7f9b87fae570 pc=0x53625f78 HEMS::INSERT-EPISODE
  23: fp=0x7f9b87fae680 pc=0x53625f78 HEMS::INSERT-EPISODE
  24: fp=0x7f9b87fae790 pc=0x53625f78 HEMS::INSERT-EPISODE
  25: fp=0x7f9b87fae8a0 pc=0x53625f78 HEMS::INSERT-EPISODE
  26: fp=0x7f9b87fae9b0 pc=0x53625f78 HEMS::INSERT-EPISODE
  27: fp=0x7f9b87faeac0 pc=0x53625f78 HEMS::INSERT-EPISODE
  28: fp=0x7f9b87faebd0 pc=0x53625f78 HEMS::INSERT-EPISODE
  29: fp=0x7f9b87faecc0 pc=0x53625f78 HEMS::INSERT-EPISODE
  30: fp=0x7f9b87faed60 pc=0x5362dbf6 HEMS::PUSH-TO-EP-BUFFER
  31: fp=0x7f9b87faee28 pc=0x52b4ac36 SB-INT::SIMPLE-EVAL-IN-LEXENV
  32: fp=0x7f9b87faee40 pc=0x52a48b33 EVAL
  33: fp=0x7f9b87faef90 pc=0x53459717 CL4PY::CL4PY
  34: fp=0x7f9b87faf058 pc=0x52b4ac36 SB-INT::SIMPLE-EVAL-IN-LEXENV
  35: fp=0x7f9b87faf080 pc=0x52c65523 SB-EXT::EVAL-TLF
  36: fp=0x7f9b87faf2a8 pc=0x52bafe04 (LABELS SB-FASL::EVAL-FORM :IN SB-INT::LOAD-AS-SOURCE)
  37: fp=0x7f9b87faf3e8 pc=0x52baf6f1 (LAMBDA (SB-KERNEL::FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-INT::LOAD-AS-SOURCE)
  38: fp=0x7f9b87faf4a0 pc=0x52b7daa5 SB-C::%DO-FORMS-FROM-INFO
  39: fp=0x7f9b87faf650 pc=0x52baf285 SB-INT::LOAD-AS-SOURCE
  40: fp=0x7f9b87faf758 pc=0x52b15566 (LABELS SB-FASL::LOAD-STREAM-1 :IN LOAD)
  41: fp=0x7f9b87faf7f0 pc=0x52c8f068 SB-FASL::CALL-WITH-LOAD-BINDINGS
  42: fp=0x7f9b87faf910 pc=0x52b15223 LOAD
  43: fp=0x7f9b87faf9f0 pc=0x52c67073 (FLET SB-IMPL::LOAD-SCRIPT :IN SB-IMPL::PROCESS-SCRIPT)
  44: fp=0x7f9b87fafa90 pc=0x52c669ca (FLET SB-UNIX::BODY :IN SB-IMPL::PROCESS-SCRIPT)
  45: fp=0x7f9b87fafb50 pc=0x52c6675a (FLET \"WITHOUT-INTERRUPTS-BODY-11\" :IN SB-IMPL::PROCESS-SCRIPT)
  46: fp=0x7f9b87fafbf0 pc=0x52c66589 SB-IMPL::PROCESS-SCRIPT
  47: fp=0x7f9b87fafdc0 pc=0x52c679b3 SB-IMPL::TOPLEVEL-INIT
  48: fp=0x7f9b87fafe60 pc=0x52f3af09 (FLET SB-UNIX::BODY :IN SB-IMPL::START-LISP)
  49: fp=0x7f9b87faff28 pc=0x52f3ad0a (FLET \"WITHOUT-INTERRUPTS-BODY-3\" :IN SB-IMPL::START-LISP)

I assume these are SBCL's last words after encountering some hash-table related bug. The good news is that this bug seems to be fixed in newer versions of SBCL. On SBCL 2.3.8, your script works just fine.

Can you confirm that switching to a newer version of SBCL fixes your problem?

dmenager commented 9 months ago

Interesting.

Thanks Marco for discovering that. I'll update my SBCL and let you know!

Le lun. 8 janv. 2024, 10:01, Marco Heisig @.***> a écrit :

I managed to reproduce your problem. I put your code snippet in a file named bug.py, and ran strace -e read -s 50000 python3 bug.py 2> log.txt to trace how the Python process communicates with its SBCL subprocess. Here is the message that breaks cl4py's parser:

0: fp=0x7f9b87fad238 pc=0x52e2f2cf SB-IMPL::HASH-TABLE-NEW-VECTORS 1: fp=0x7f9b87fad2a0 pc=0x52e2fa76 SB-IMPL::GROW-HASH-TABLE 2: fp=0x7f9b87fad308 pc=0x52ba0cb9 (FLET SB-IMPL::INSERT-AT :IN \"SYS:SRC;CODE;TARGET-HASH-TABLE.LISP\") 3: fp=0x7f9b87fad370 pc=0x52ba01d1 SB-IMPL::PUTHASH/EQL 4: fp=0x7f9b87fad398 pc=0x52a1667c SB-KERNEL::%PUTHASH 5: fp=0x7f9b87fad458 pc=0x535de3ef HEMS::UPDATE-CPD-RULES 6: fp=0x7f9b87fad520 pc=0x535f0069 HEMS::FACTOR-FILTER 7: fp=0x7f9b87fad618 pc=0x5362137a HEMS::COMBINE-STATES 8: fp=0x7f9b87fad6e8 pc=0x53622901 HEMS::COMBINE 9: fp=0x7f9b87fad7a0 pc=0x5362474a HEMS::EP-MERGE 10: fp=0x7f9b87fad8b0 pc=0x53625b5f HEMS::INSERT-EPISODE 11: fp=0x7f9b87fad9c0 pc=0x53625f78 HEMS::INSERT-EPISODE 12: fp=0x7f9b87fadad0 pc=0x53625f78 HEMS::INSERT-EPISODE 13: fp=0x7f9b87fadbe0 pc=0x53625f78 HEMS::INSERT-EPISODE 14: fp=0x7f9b87fadcf0 pc=0x53625f78 HEMS::INSERT-EPISODE 15: fp=0x7f9b87fade00 pc=0x53625f78 HEMS::INSERT-EPISODE 16: fp=0x7f9b87fadf10 pc=0x53625f78 HEMS::INSERT-EPISODE 17: fp=0x7f9b87fae020 pc=0x53625f78 HEMS::INSERT-EPISODE 18: fp=0x7f9b87fae130 pc=0x53625f78 HEMS::INSERT-EPISODE 19: fp=0x7f9b87fae240 pc=0x53625f78 HEMS::INSERT-EPISODE 20: fp=0x7f9b87fae350 pc=0x53625f78 HEMS::INSERT-EPISODE 21: fp=0x7f9b87fae460 pc=0x53625f78 HEMS::INSERT-EPISODE 22: fp=0x7f9b87fae570 pc=0x53625f78 HEMS::INSERT-EPISODE 23: fp=0x7f9b87fae680 pc=0x53625f78 HEMS::INSERT-EPISODE 24: fp=0x7f9b87fae790 pc=0x53625f78 HEMS::INSERT-EPISODE 25: fp=0x7f9b87fae8a0 pc=0x53625f78 HEMS::INSERT-EPISODE 26: fp=0x7f9b87fae9b0 pc=0x53625f78 HEMS::INSERT-EPISODE 27: fp=0x7f9b87faeac0 pc=0x53625f78 HEMS::INSERT-EPISODE 28: fp=0x7f9b87faebd0 pc=0x53625f78 HEMS::INSERT-EPISODE 29: fp=0x7f9b87faecc0 pc=0x53625f78 HEMS::INSERT-EPISODE 30: fp=0x7f9b87faed60 pc=0x5362dbf6 HEMS::PUSH-TO-EP-BUFFER 31: fp=0x7f9b87faee28 pc=0x52b4ac36 SB-INT::SIMPLE-EVAL-IN-LEXENV 32: fp=0x7f9b87faee40 pc=0x52a48b33 EVAL 33: fp=0x7f9b87faef90 pc=0x53459717 CL4PY::CL4PY 34: fp=0x7f9b87faf058 pc=0x52b4ac36 SB-INT::SIMPLE-EVAL-IN-LEXENV 35: fp=0x7f9b87faf080 pc=0x52c65523 SB-EXT::EVAL-TLF 36: fp=0x7f9b87faf2a8 pc=0x52bafe04 (LABELS SB-FASL::EVAL-FORM :IN SB-INT::LOAD-AS-SOURCE) 37: fp=0x7f9b87faf3e8 pc=0x52baf6f1 (LAMBDA (SB-KERNEL::FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-INT::LOAD-AS-SOURCE) 38: fp=0x7f9b87faf4a0 pc=0x52b7daa5 SB-C::%DO-FORMS-FROM-INFO 39: fp=0x7f9b87faf650 pc=0x52baf285 SB-INT::LOAD-AS-SOURCE 40: fp=0x7f9b87faf758 pc=0x52b15566 (LABELS SB-FASL::LOAD-STREAM-1 :IN LOAD) 41: fp=0x7f9b87faf7f0 pc=0x52c8f068 SB-FASL::CALL-WITH-LOAD-BINDINGS 42: fp=0x7f9b87faf910 pc=0x52b15223 LOAD 43: fp=0x7f9b87faf9f0 pc=0x52c67073 (FLET SB-IMPL::LOAD-SCRIPT :IN SB-IMPL::PROCESS-SCRIPT) 44: fp=0x7f9b87fafa90 pc=0x52c669ca (FLET SB-UNIX::BODY :IN SB-IMPL::PROCESS-SCRIPT) 45: fp=0x7f9b87fafb50 pc=0x52c6675a (FLET \"WITHOUT-INTERRUPTS-BODY-11\" :IN SB-IMPL::PROCESS-SCRIPT) 46: fp=0x7f9b87fafbf0 pc=0x52c66589 SB-IMPL::PROCESS-SCRIPT 47: fp=0x7f9b87fafdc0 pc=0x52c679b3 SB-IMPL::TOPLEVEL-INIT 48: fp=0x7f9b87fafe60 pc=0x52f3af09 (FLET SB-UNIX::BODY :IN SB-IMPL::START-LISP) 49: fp=0x7f9b87faff28 pc=0x52f3ad0a (FLET \"WITHOUT-INTERRUPTS-BODY-3\" :IN SB-IMPL::START-LISP)

I assume these are SBCL's last words after encountering some hash-table related bug. The good news is that this bug seems to be fixed in newer versions of SBCL. On SBCL 2.3.8, your script works just fine.

Can you confirm that switching to a newer version of SBCL fixes your problem?

— Reply to this email directly, view it on GitHub https://github.com/marcoheisig/cl4py/issues/24#issuecomment-1881358908, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA46I5BA2ATHSKV53KOSX6TYNQJ7BAVCNFSM6AAAAABBN3JM7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBRGM2TQOJQHA . You are receiving this because you authored the thread.Message ID: @.***>

dmenager commented 9 months ago

Hi Marco,

So, I can confirm that I still observe the problem on my end after updating SBCL. Please pull from the HEMS master branch (for efficiency improvement) and run again. I am running SBCL version 2.4.0.

Le lun. 8 janv. 2024, 10:08, David Menager @.***> a écrit :

Interesting.

Thanks Marco for discovering that. I'll update my SBCL and let you know!

D

Le lun. 8 janv. 2024, 10:01, Marco Heisig @.***> a écrit :

I managed to reproduce your problem. I put your code snippet in a file named bug.py, and ran strace -e read -s 50000 python3 bug.py 2> log.txt to trace how the Python process communicates with its SBCL subprocess. Here is the message that breaks cl4py's parser:

0: fp=0x7f9b87fad238 pc=0x52e2f2cf SB-IMPL::HASH-TABLE-NEW-VECTORS 1: fp=0x7f9b87fad2a0 pc=0x52e2fa76 SB-IMPL::GROW-HASH-TABLE 2: fp=0x7f9b87fad308 pc=0x52ba0cb9 (FLET SB-IMPL::INSERT-AT :IN \"SYS:SRC;CODE;TARGET-HASH-TABLE.LISP\") 3: fp=0x7f9b87fad370 pc=0x52ba01d1 SB-IMPL::PUTHASH/EQL 4: fp=0x7f9b87fad398 pc=0x52a1667c SB-KERNEL::%PUTHASH 5: fp=0x7f9b87fad458 pc=0x535de3ef HEMS::UPDATE-CPD-RULES 6: fp=0x7f9b87fad520 pc=0x535f0069 HEMS::FACTOR-FILTER 7: fp=0x7f9b87fad618 pc=0x5362137a HEMS::COMBINE-STATES 8: fp=0x7f9b87fad6e8 pc=0x53622901 HEMS::COMBINE 9: fp=0x7f9b87fad7a0 pc=0x5362474a HEMS::EP-MERGE 10: fp=0x7f9b87fad8b0 pc=0x53625b5f HEMS::INSERT-EPISODE 11: fp=0x7f9b87fad9c0 pc=0x53625f78 HEMS::INSERT-EPISODE 12: fp=0x7f9b87fadad0 pc=0x53625f78 HEMS::INSERT-EPISODE 13: fp=0x7f9b87fadbe0 pc=0x53625f78 HEMS::INSERT-EPISODE 14: fp=0x7f9b87fadcf0 pc=0x53625f78 HEMS::INSERT-EPISODE 15: fp=0x7f9b87fade00 pc=0x53625f78 HEMS::INSERT-EPISODE 16: fp=0x7f9b87fadf10 pc=0x53625f78 HEMS::INSERT-EPISODE 17: fp=0x7f9b87fae020 pc=0x53625f78 HEMS::INSERT-EPISODE 18: fp=0x7f9b87fae130 pc=0x53625f78 HEMS::INSERT-EPISODE 19: fp=0x7f9b87fae240 pc=0x53625f78 HEMS::INSERT-EPISODE 20: fp=0x7f9b87fae350 pc=0x53625f78 HEMS::INSERT-EPISODE 21: fp=0x7f9b87fae460 pc=0x53625f78 HEMS::INSERT-EPISODE 22: fp=0x7f9b87fae570 pc=0x53625f78 HEMS::INSERT-EPISODE 23: fp=0x7f9b87fae680 pc=0x53625f78 HEMS::INSERT-EPISODE 24: fp=0x7f9b87fae790 pc=0x53625f78 HEMS::INSERT-EPISODE 25: fp=0x7f9b87fae8a0 pc=0x53625f78 HEMS::INSERT-EPISODE 26: fp=0x7f9b87fae9b0 pc=0x53625f78 HEMS::INSERT-EPISODE 27: fp=0x7f9b87faeac0 pc=0x53625f78 HEMS::INSERT-EPISODE 28: fp=0x7f9b87faebd0 pc=0x53625f78 HEMS::INSERT-EPISODE 29: fp=0x7f9b87faecc0 pc=0x53625f78 HEMS::INSERT-EPISODE 30: fp=0x7f9b87faed60 pc=0x5362dbf6 HEMS::PUSH-TO-EP-BUFFER 31: fp=0x7f9b87faee28 pc=0x52b4ac36 SB-INT::SIMPLE-EVAL-IN-LEXENV 32: fp=0x7f9b87faee40 pc=0x52a48b33 EVAL 33: fp=0x7f9b87faef90 pc=0x53459717 CL4PY::CL4PY 34: fp=0x7f9b87faf058 pc=0x52b4ac36 SB-INT::SIMPLE-EVAL-IN-LEXENV 35: fp=0x7f9b87faf080 pc=0x52c65523 SB-EXT::EVAL-TLF 36: fp=0x7f9b87faf2a8 pc=0x52bafe04 (LABELS SB-FASL::EVAL-FORM :IN SB-INT::LOAD-AS-SOURCE) 37: fp=0x7f9b87faf3e8 pc=0x52baf6f1 (LAMBDA (SB-KERNEL::FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-INT::LOAD-AS-SOURCE) 38: fp=0x7f9b87faf4a0 pc=0x52b7daa5 SB-C::%DO-FORMS-FROM-INFO 39: fp=0x7f9b87faf650 pc=0x52baf285 SB-INT::LOAD-AS-SOURCE 40: fp=0x7f9b87faf758 pc=0x52b15566 (LABELS SB-FASL::LOAD-STREAM-1 :IN LOAD) 41: fp=0x7f9b87faf7f0 pc=0x52c8f068 SB-FASL::CALL-WITH-LOAD-BINDINGS 42: fp=0x7f9b87faf910 pc=0x52b15223 LOAD 43: fp=0x7f9b87faf9f0 pc=0x52c67073 (FLET SB-IMPL::LOAD-SCRIPT :IN SB-IMPL::PROCESS-SCRIPT) 44: fp=0x7f9b87fafa90 pc=0x52c669ca (FLET SB-UNIX::BODY :IN SB-IMPL::PROCESS-SCRIPT) 45: fp=0x7f9b87fafb50 pc=0x52c6675a (FLET \"WITHOUT-INTERRUPTS-BODY-11\" :IN SB-IMPL::PROCESS-SCRIPT) 46: fp=0x7f9b87fafbf0 pc=0x52c66589 SB-IMPL::PROCESS-SCRIPT 47: fp=0x7f9b87fafdc0 pc=0x52c679b3 SB-IMPL::TOPLEVEL-INIT 48: fp=0x7f9b87fafe60 pc=0x52f3af09 (FLET SB-UNIX::BODY :IN SB-IMPL::START-LISP) 49: fp=0x7f9b87faff28 pc=0x52f3ad0a (FLET \"WITHOUT-INTERRUPTS-BODY-3\" :IN SB-IMPL::START-LISP)

I assume these are SBCL's last words after encountering some hash-table related bug. The good news is that this bug seems to be fixed in newer versions of SBCL. On SBCL 2.3.8, your script works just fine.

Can you confirm that switching to a newer version of SBCL fixes your problem?

— Reply to this email directly, view it on GitHub https://github.com/marcoheisig/cl4py/issues/24#issuecomment-1881358908, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA46I5BA2ATHSKV53KOSX6TYNQJ7BAVCNFSM6AAAAABBN3JM7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBRGM2TQOJQHA . You are receiving this because you authored the thread.Message ID: @.***>

dmenager commented 9 months ago

I ran the same command you showed and generated a log file which I'm attaching here.

D log.txt https://drive.google.com/file/d/19tUUrmofKlB5xG1uyTYOmwrdCS9eGCoU/view?usp=drive_web

On Tue, Jan 9, 2024 at 10:32 AM David Menager @.***> wrote:

Hi Marco,

So, I can confirm that I still observe the problem on my end after updating SBCL. Please pull from the HEMS master branch (for efficiency improvement) and run again. I am running SBCL version 2.4.0.

D

Le lun. 8 janv. 2024, 10:08, David Menager @.***> a écrit :

Interesting.

Thanks Marco for discovering that. I'll update my SBCL and let you know!

D

Le lun. 8 janv. 2024, 10:01, Marco Heisig @.***> a écrit :

I managed to reproduce your problem. I put your code snippet in a file named bug.py, and ran strace -e read -s 50000 python3 bug.py 2> log.txt to trace how the Python process communicates with its SBCL subprocess. Here is the message that breaks cl4py's parser:

0: fp=0x7f9b87fad238 pc=0x52e2f2cf SB-IMPL::HASH-TABLE-NEW-VECTORS 1: fp=0x7f9b87fad2a0 pc=0x52e2fa76 SB-IMPL::GROW-HASH-TABLE 2: fp=0x7f9b87fad308 pc=0x52ba0cb9 (FLET SB-IMPL::INSERT-AT :IN \"SYS:SRC;CODE;TARGET-HASH-TABLE.LISP\") 3: fp=0x7f9b87fad370 pc=0x52ba01d1 SB-IMPL::PUTHASH/EQL 4: fp=0x7f9b87fad398 pc=0x52a1667c SB-KERNEL::%PUTHASH 5: fp=0x7f9b87fad458 pc=0x535de3ef HEMS::UPDATE-CPD-RULES 6: fp=0x7f9b87fad520 pc=0x535f0069 HEMS::FACTOR-FILTER 7: fp=0x7f9b87fad618 pc=0x5362137a HEMS::COMBINE-STATES 8: fp=0x7f9b87fad6e8 pc=0x53622901 HEMS::COMBINE 9: fp=0x7f9b87fad7a0 pc=0x5362474a HEMS::EP-MERGE 10: fp=0x7f9b87fad8b0 pc=0x53625b5f HEMS::INSERT-EPISODE 11: fp=0x7f9b87fad9c0 pc=0x53625f78 HEMS::INSERT-EPISODE 12: fp=0x7f9b87fadad0 pc=0x53625f78 HEMS::INSERT-EPISODE 13: fp=0x7f9b87fadbe0 pc=0x53625f78 HEMS::INSERT-EPISODE 14: fp=0x7f9b87fadcf0 pc=0x53625f78 HEMS::INSERT-EPISODE 15: fp=0x7f9b87fade00 pc=0x53625f78 HEMS::INSERT-EPISODE 16: fp=0x7f9b87fadf10 pc=0x53625f78 HEMS::INSERT-EPISODE 17: fp=0x7f9b87fae020 pc=0x53625f78 HEMS::INSERT-EPISODE 18: fp=0x7f9b87fae130 pc=0x53625f78 HEMS::INSERT-EPISODE 19: fp=0x7f9b87fae240 pc=0x53625f78 HEMS::INSERT-EPISODE 20: fp=0x7f9b87fae350 pc=0x53625f78 HEMS::INSERT-EPISODE 21: fp=0x7f9b87fae460 pc=0x53625f78 HEMS::INSERT-EPISODE 22: fp=0x7f9b87fae570 pc=0x53625f78 HEMS::INSERT-EPISODE 23: fp=0x7f9b87fae680 pc=0x53625f78 HEMS::INSERT-EPISODE 24: fp=0x7f9b87fae790 pc=0x53625f78 HEMS::INSERT-EPISODE 25: fp=0x7f9b87fae8a0 pc=0x53625f78 HEMS::INSERT-EPISODE 26: fp=0x7f9b87fae9b0 pc=0x53625f78 HEMS::INSERT-EPISODE 27: fp=0x7f9b87faeac0 pc=0x53625f78 HEMS::INSERT-EPISODE 28: fp=0x7f9b87faebd0 pc=0x53625f78 HEMS::INSERT-EPISODE 29: fp=0x7f9b87faecc0 pc=0x53625f78 HEMS::INSERT-EPISODE 30: fp=0x7f9b87faed60 pc=0x5362dbf6 HEMS::PUSH-TO-EP-BUFFER 31: fp=0x7f9b87faee28 pc=0x52b4ac36 SB-INT::SIMPLE-EVAL-IN-LEXENV 32: fp=0x7f9b87faee40 pc=0x52a48b33 EVAL 33: fp=0x7f9b87faef90 pc=0x53459717 CL4PY::CL4PY 34: fp=0x7f9b87faf058 pc=0x52b4ac36 SB-INT::SIMPLE-EVAL-IN-LEXENV 35: fp=0x7f9b87faf080 pc=0x52c65523 SB-EXT::EVAL-TLF 36: fp=0x7f9b87faf2a8 pc=0x52bafe04 (LABELS SB-FASL::EVAL-FORM :IN SB-INT::LOAD-AS-SOURCE) 37: fp=0x7f9b87faf3e8 pc=0x52baf6f1 (LAMBDA (SB-KERNEL::FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-INT::LOAD-AS-SOURCE) 38: fp=0x7f9b87faf4a0 pc=0x52b7daa5 SB-C::%DO-FORMS-FROM-INFO 39: fp=0x7f9b87faf650 pc=0x52baf285 SB-INT::LOAD-AS-SOURCE 40: fp=0x7f9b87faf758 pc=0x52b15566 (LABELS SB-FASL::LOAD-STREAM-1 :IN LOAD) 41: fp=0x7f9b87faf7f0 pc=0x52c8f068 SB-FASL::CALL-WITH-LOAD-BINDINGS 42: fp=0x7f9b87faf910 pc=0x52b15223 LOAD 43: fp=0x7f9b87faf9f0 pc=0x52c67073 (FLET SB-IMPL::LOAD-SCRIPT :IN SB-IMPL::PROCESS-SCRIPT) 44: fp=0x7f9b87fafa90 pc=0x52c669ca (FLET SB-UNIX::BODY :IN SB-IMPL::PROCESS-SCRIPT) 45: fp=0x7f9b87fafb50 pc=0x52c6675a (FLET \"WITHOUT-INTERRUPTS-BODY-11\" :IN SB-IMPL::PROCESS-SCRIPT) 46: fp=0x7f9b87fafbf0 pc=0x52c66589 SB-IMPL::PROCESS-SCRIPT 47: fp=0x7f9b87fafdc0 pc=0x52c679b3 SB-IMPL::TOPLEVEL-INIT 48: fp=0x7f9b87fafe60 pc=0x52f3af09 (FLET SB-UNIX::BODY :IN SB-IMPL::START-LISP) 49: fp=0x7f9b87faff28 pc=0x52f3ad0a (FLET \"WITHOUT-INTERRUPTS-BODY-3\" :IN SB-IMPL::START-LISP)

I assume these are SBCL's last words after encountering some hash-table related bug. The good news is that this bug seems to be fixed in newer versions of SBCL. On SBCL 2.3.8, your script works just fine.

Can you confirm that switching to a newer version of SBCL fixes your problem?

— Reply to this email directly, view it on GitHub https://github.com/marcoheisig/cl4py/issues/24#issuecomment-1881358908, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA46I5BA2ATHSKV53KOSX6TYNQJ7BAVCNFSM6AAAAABBN3JM7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBRGM2TQOJQHA . You are receiving this because you authored the thread.Message ID: @.***>

marcoheisig commented 9 months ago

I pulled HEMS from your master branch an ran again, and the problem is still gone for me. Are you sure cl4py is actually invoking sbcl 2.4.0 and not an older version? (When in doubt, check cl.lisp_implementation_version() in cl4py)

With the old version of sbcl, your code reliably crashes at episode 28. With the new version, I just reached episode 270 until I manually interrupted the program.

Either way, it seems to me that cl4py is not the problem. Either your code does something unsafe with hash tables, or sbcl has a bug.

PS: I couldn't look at your logfile because I don't have a Google account.

dmenager commented 9 months ago

Can I kindly ask you to let the system run through without interrupting it? There are only about 550 episodes.

Also, I tried to run cl.lisp_implementation_version(), but never got it working. I assume I need to run lisp.find_package('CL'), which unfortunately hangs forever for me. Not sure why though.

On Tue, Jan 9, 2024 at 12:34 PM Marco Heisig @.***> wrote:

I pulled HEMS from your master branch an ran again, and the problem is still gone for me. Are you sure cl4py is actually invoking sbcl 2.4.0 and not an older version? (When in doubt, check cl.lisp_implementation_version() in cl4py)

With the old version of sbcl, your code reliably crashes at episode 28. With the new version, I just reached episode 270 until I manually interrupted the program.

Either way, it seems to me that cl4py is not the problem. Either your code does something unsafe with hash tables, or sbcl has a bug.

PS: I couldn't look at your logfile because I don't have a Google account.

— Reply to this email directly, view it on GitHub https://github.com/marcoheisig/cl4py/issues/24#issuecomment-1883495195, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA46I5HYFUQC2LFPOTFS4TLYNV5SHAVCNFSM6AAAAABBN3JM7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBTGQ4TKMJZGU . You are receiving this because you authored the thread.Message ID: @.***>

marcoheisig commented 9 months ago

Can I kindly ask you to let the system run through without interrupting it?

Done. Reached episode 566 just fine. Here are the last lines of its output:

Adding new episode.
inserting EPISODE-6512397
564

Adding new episode.
inserting EPISODE-6535858
565

Adding new episode.
inserting EPISODE-6559064
566

Adding new episode.
inserting EPISODE-6582208

This snippet of Python code should give you the Lisp version:

import cl4py
lisp = cl4py.Lisp()
cl = lisp.find_package('CL')
cl.lisp_implementation_version()

That code really shouldn't hang. If it does, consider opening another Github issue.

My suspicion would be that you have multiple versions of sbcl installed, but cl4py invokes the wrong one. In that case, you should create your Lisp process like this to explicitly specify the correct sbcl executable:

lisp = cl4py.Lisp(cmd=('/PATH/TO/RECENT/sbcl', '--script'))

My other guess would be that you are now running out of heap space. In that case you could try adding something like '--dynamic-space-size', '20000' to the above command.

If that doesn't solve the issue I am out of ideas, mostly because I cannot reproduce your problem anymore.

dmenager commented 8 months ago

Hi Marco,

I took some time away from this project to handle other things, but now I'm back focusing on this. Unfortunately, I'm still getting errors. I'm using python 3.7. When I run lisp = cl4py.Lisp(cmd=('/PATH/TO/SBCL', '--script')) I get the same error as before, when I increase the dynamic space size I get an EOFError, strangely enough. It must be that something is off with my system, as you're not getting the same issues as me and you've run that code.

As far as opening a new issue, I want to make sure at least that the problem isn't with my system configuration first. Once I am reasonably confident that the problem is really with the library, I'll make an issue about find_package hanging indefinitely.

I would really like to connect with you on a call so we can troubleshoot this together and you can see what I'm doing. Does this sound alright with you? If so, then, please let me know when you're available and I'll reach out with my work email.

On Tue, Jan 9, 2024 at 6:28 PM Marco Heisig @.***> wrote:

Can I kindly ask you to let the system run through without interrupting it?

Done. Reached episode 566 just fine. Here are the last lines of its output:

Adding new episode. inserting EPISODE-6512397 564

Adding new episode. inserting EPISODE-6535858 565

Adding new episode. inserting EPISODE-6559064 566

Adding new episode. inserting EPISODE-6582208

This snippet of Python code should give you the Lisp version:

import cl4pylisp = cl4py.Lisp()cl = lisp.find_package('CL')cl.lisp_implementation_version()

That code really shouldn't hang. If it does, consider opening another Github issue.

My suspicion would be that you have multiple versions of sbcl installed, but cl4py invokes the wrong one. In that case, you should create your Lisp process like this to explicitly specify the correct sbcl executable:

lisp = cl4py.Lisp(cmd=('/PATH/TO/RECENT/sbcl', '--script'))

My other guess would be that you are now running out of heap space. In that case you could try adding something like '--dynamic-space-size', '20000' to the above command.

If that doesn't solve the issue I am out of ideas, mostly because I cannot reproduce your problem anymore.

— Reply to this email directly, view it on GitHub https://github.com/marcoheisig/cl4py/issues/24#issuecomment-1883956408, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA46I5BFVEO6JXOQXFR37I3YNXHBVAVCNFSM6AAAAABBN3JM7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBTHE2TMNBQHA . You are receiving this because you authored the thread.Message ID: @.***>

dmenager commented 8 months ago

Hi Marco,

You were right! I was running out of heap space. I hadn't realized that the ordering of the arguments mattered when starting the lisp subprocess. So the correct command is:

lisp = cl4py.Lisp(cmd=('/usr/local/bin/sbcl', '--dynamic-space-size', '30000', '--script'), quicklisp=True, backtrace=True)

That fixed my issue.

Thanks for your help!