search

internetarchive / openlibrary

One webpage for every book ever published!
https://openlibrary.org
GNU Affero General Public License v3.0
5.26k stars 1.4k forks source link

Python 3.12: tokenize.TokenError: ('unterminated string literal #8694

Closed scottbarnes closed 8 months ago

scottbarnes commented 10 months ago

Currently there are a few errors when running tests (and indeed, loading... any page on the site) when using Python 3.12.

Edit: this appears to be a web.py problem that will likely prevent Open Library from using Python 3.12 until the issue in web.py is fixed. An issue has been opened for web.py at webpy/webpy#784.

I am not sure what change in 3.12 is responsible. Perhaps PEP 701: Syntactic formalization of f-strings, though at first glace the code that's causing the errors has no f-strings, but it is at least related to string parsing.

Evidence / Screenshot (if possible)

pytest error:

___________________________________________________________________________________________________________________ ERROR collecting openlibrary/tests/data/test_dump.py ___________________________________________________________________________________________________________________
/home/openlibrary/.local/lib/python3.12/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
/home/openlibrary/.local/lib/python3.12/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/home/openlibrary/.local/lib/python3.12/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
/home/openlibrary/.local/lib/python3.12/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/home/openlibrary/.local/lib/python3.12/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
/home/openlibrary/.local/lib/python3.12/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
/home/openlibrary/.local/lib/python3.12/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/home/openlibrary/.local/lib/python3.12/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/usr/local/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:935: in _load_unlocked
    ???
/home/openlibrary/.local/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:186: in exec_module
    exec(co, module.__dict__)
openlibrary/tests/data/test_dump.py:3: in <module>
    from openlibrary.data.dump import print_dump, pgdecode
openlibrary/data/__init__.py:5: in <module>
    from openlibrary.data.dump import pgdecode
openlibrary/data/dump.py:22: in <module>
    from openlibrary.data.sitemap import generate_html_index, generate_sitemaps
openlibrary/data/sitemap.py:67: in <module>
    t_html_sitemap = t(
/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py:932: in __init__
    code = self.compile_template(text, filename)
/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py:1003: in compile_template
    code = Template.generate_code(
/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py:988: in generate_code
    rootnode = parser.parse(text, filename)
/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py:87: in parse
    suite = self.read_suite(text)
/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py:175: in read_suite
    section, text = self.read_section(text)
/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py:126: in read_section
    return self.readline(text)
/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py:198: in readline
    node, line = self.read_node(line)
/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py:224: in read_node
    return self.read_expr(text, escape=escape)
/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py:362: in read_expr
    simple_expr()
/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py:271: in simple_expr
    extended_expr()
/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py:277: in extended_expr
    lookahead = tokens.lookahead()
/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py:338: in lookahead
    self.items.append(self._next())
/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py:343: in _next
    return next(self.iteritems)
/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py:318: in get_tokens
    for t in tokenize.generate_tokens(readline):
/usr/local/lib/python3.12/tokenize.py:543: in _generate_tokens_from_c_tokenizer
    raise TokenError(msg, (e.lineno, e.offset)) from None
E   tokenize.TokenError: ('unterminated string literal (detected at line 1)', (1, 5))

Similarly, visiting any page that should load a template shows something like the following in the Docker console (note the multiple template parsing failures):

web-1           | 0.0 (1): SELECT * FROM store WHERE key='account/openlibrary'
web-1           | 0.0 (2): SELECT * FROM store WHERE key='counts-2023-12-08'
web-1           | 0.0 (3): SELECT * FROM store WHERE key='counts-2023-12-09'
web-1           | 0.0 (4): SELECT * FROM store WHERE key='counts-2023-12-10'
web-1           | 0.0 (5): SELECT * FROM store WHERE key='counts-2023-12-11'
web-1           | 0.0 (6): SELECT * FROM store WHERE key='counts-2023-12-12'
web-1           | 0.0 (7): SELECT * FROM store WHERE key='counts-2023-12-13'
web-1           | 0.0 (8): SELECT * FROM store WHERE key='counts-2023-12-14'
web-1           | 0.0 (9): SELECT * FROM store WHERE key='counts-2023-12-15'
web-1           | 0.0 (10): SELECT * FROM store WHERE key='counts-2023-12-16'
web-1           | 0.0 (11): SELECT * FROM store WHERE key='counts-2023-12-17'
web-1           | 0.0 (12): SELECT * FROM store WHERE key='counts-2023-12-18'
web-1           | 0.0 (13): SELECT * FROM store WHERE key='counts-2023-12-19'
web-1           | 0.0 (14): SELECT * FROM store WHERE key='counts-2023-12-20'
web-1           | 0.0 (15): SELECT * FROM store WHERE key='counts-2023-12-21'
web-1           | 0.0 (16): SELECT * FROM store WHERE key='counts-2023-12-22'
web-1           | 0.0 (17): SELECT * FROM store WHERE key='counts-2023-12-23'
web-1           | 0.0 (18): SELECT * FROM store WHERE key='counts-2023-12-24'
web-1           | 0.0 (19): SELECT * FROM store WHERE key='counts-2023-12-25'
web-1           | 0.0 (20): SELECT * FROM store WHERE key='counts-2023-12-26'
web-1           | 0.0 (21): SELECT * FROM store WHERE key='counts-2023-12-27'
web-1           | 0.0 (22): SELECT * FROM store WHERE key='counts-2023-12-28'
web-1           | 0.0 (23): SELECT * FROM store WHERE key='counts-2023-12-29'
web-1           | 0.0 (24): SELECT * FROM store WHERE key='counts-2023-12-30'
web-1           | 0.0 (25): SELECT * FROM store WHERE key='counts-2023-12-31'
web-1           | 0.0 (26): SELECT * FROM store WHERE key='counts-2024-01-01'
web-1           | 0.0 (27): SELECT * FROM store WHERE key='counts-2024-01-02'
web-1           | 0.0 (28): SELECT * FROM store WHERE key='counts-2024-01-03'
web-1           | 0.0 (29): SELECT * FROM store WHERE key='counts-2024-01-04'
web-1           | 0.0 (30): SELECT * FROM store WHERE key='counts-2024-01-05'
web-1           | 0.0 (31): SELECT * FROM store WHERE key='counts-2024-01-06'
web-1           | Traceback (most recent call last):
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/application.py", line 276, in process
web-1           |     return self.handle()
web-1           |            ^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/application.py", line 267, in handle
web-1           |     return self._delegate(fn, self.fvars, args)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/application.py", line 509, in _delegate
web-1           |     return handle_class(cls)
web-1           |            ^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/application.py", line 487, in handle_class
web-1           |     return tocall(*args)
web-1           |            ^^^^^^^^^^^^^
web-1           |   File "/openlibrary/infogami/utils/app.py", line 197, in <lambda>
web-1           |     HEAD = GET = POST = PUT = DELETE = lambda self: delegate()
web-1           |                                                     ^^^^^^^^^^
web-1           |   File "/openlibrary/infogami/utils/app.py", line 217, in delegate
web-1           |     return getattr(cls(), method)(*args)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/openlibrary/openlibrary/plugins/openlibrary/home.py", line 99, in GET
web-1           |     cached_homepage = get_cached_homepage()
web-1           |                       ^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/openlibrary/openlibrary/plugins/openlibrary/home.py", line 68, in get_cached_homepage
web-1           |     page = mc()
web-1           |            ^^^^
web-1           |   File "/openlibrary/openlibrary/core/cache.py", line 124, in __call__
web-1           |     value, t = self.update(*args, **kw)
web-1           |                ^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/openlibrary/openlibrary/core/cache.py", line 164, in update
web-1           |     value = self.f(*args, **kw)
web-1           |             ^^^^^^^^^^^^^^^^^^^
web-1           |   File "/openlibrary/openlibrary/plugins/openlibrary/home.py", line 51, in get_homepage
web-1           |     page = render_template("home/index", stats=stats, blog_posts=blog_posts)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/openlibrary/infogami/utils/template.py", line 219, in render_template
web-1           |     return get_template(name)(*a, **kw)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/openlibrary/infogami/utils/template.py", line 116, in <lambda>
web-1           |     return lambda *a, **kw: saferender(templates, *a, **kw)
web-1           |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/openlibrary/infogami/utils/template.py", line 139, in g
web-1           |     return f(*a, **kw)
web-1           |            ^^^^^^^^^^^
web-1           |   File "/openlibrary/infogami/utils/template.py", line 172, in saferender
web-1           |     delegate.register_exception()
web-1           |   File "/openlibrary/infogami/utils/delegate.py", line 222, in register_exception
web-1           |     h()
web-1           |   File "/openlibrary/openlibrary/plugins/openlibrary/code.py", line 931, in save_error
web-1           |     error = web.safestr(web.djangoerror())
web-1           |                         ^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/debugerror.py", line 307, in djangoerror
web-1           |     djangoerror_r = Template(djangoerror_t, filename=__file__, filter=websafe)
web-1           |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 932, in __init__
web-1           |     code = self.compile_template(text, filename)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 1003, in compile_template
web-1           |     code = Template.generate_code(
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 988, in generate_code
web-1           |     rootnode = parser.parse(text, filename)
web-1           |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 87, in parse
web-1           |     suite = self.read_suite(text)
web-1           |             ^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 175, in read_suite
web-1           |     section, text = self.read_section(text)
web-1           |                     ^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 119, in read_section
web-1           |     return self.read_block_section(text2, begin_indent)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 477, in read_block_section
web-1           |     return self.create_block_node(keyword, stmt, block, begin_indent), text
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 481, in create_block_node
web-1           |     return self.statement_nodes[keyword](stmt, block, begin_indent)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 692, in __init__
web-1           |     BlockNode.__init__(self, *a, **kw)
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 626, in __init__
web-1           |     self.suite = Parser().read_suite(block)
web-1           |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 175, in read_suite
web-1           |     section, text = self.read_section(text)
web-1           |                     ^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 119, in read_section
web-1           |     return self.read_block_section(text2, begin_indent)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 477, in read_block_section
web-1           |     return self.create_block_node(keyword, stmt, block, begin_indent), text
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 481, in create_block_node
web-1           |     return self.statement_nodes[keyword](stmt, block, begin_indent)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 626, in __init__
web-1           |     self.suite = Parser().read_suite(block)
web-1           |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 175, in read_suite
web-1           |     section, text = self.read_section(text)
web-1           |                     ^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 126, in read_section
web-1           |     return self.readline(text)
web-1           |            ^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 198, in readline
web-1           |     node, line = self.read_node(line)
web-1           |                  ^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 224, in read_node
web-1           |     return self.read_expr(text, escape=escape)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 362, in read_expr
web-1           |     simple_expr()
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 271, in simple_expr
web-1           |     extended_expr()
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 277, in extended_expr
web-1           |     lookahead = tokens.lookahead()
web-1           |                 ^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 338, in lookahead
web-1           |     self.items.append(self._next())
web-1           |                       ^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 343, in _next
web-1           |     return next(self.iteritems)
web-1           |            ^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 318, in get_tokens
web-1           |     for t in tokenize.generate_tokens(readline):
web-1           |   File "/usr/local/lib/python3.12/tokenize.py", line 543, in _generate_tokens_from_c_tokenizer
web-1           |     raise TokenError(msg, (e.lineno, e.offset)) from None
web-1           | tokenize.TokenError: ('unterminated string literal (detected at line 1)', (1, 4))
web-1           |

*** full error ommitted because it was too long for a GitHub issue body, but it's more of the same, but with various templates ***

web-1           | [2024-01-06 05:40:12 +0000] [10] [ERROR] Error handling request /
web-1           | Traceback (most recent call last):
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/gunicorn/workers/sync.py", line 136, in handle
web-1           |     self.handle_request(listener, req, client, addr)
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/gunicorn/workers/sync.py", line 179, in handle_request
web-1           |     respiter = self.wsgi(environ, resp.start_response)
web-1           |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/httpserver.py", line 258, in __call__
web-1           |     return self.app(environ, start_response)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/openlibrary/openlibrary/core/middleware.py", line 46, in __call__
web-1           |     data = self.app(environ, new_start_response)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/openlibrary/scripts/openlibrary-server", line 129, in wrapper
web-1           |     return app(environ, start_response)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/application.py", line 315, in wsgi
web-1           |     result = self.handle_with_processors()
web-1           |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/application.py", line 286, in handle_with_processors
web-1           |     return process(self.processors)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/application.py", line 283, in process
web-1           |     raise self.internalerror()
web-1           |           ^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/openlibrary/openlibrary/plugins/openlibrary/code.py", line 942, in internalerror
web-1           |     name = save_error()
web-1           |            ^^^^^^^^^^^^
web-1           |   File "/openlibrary/openlibrary/plugins/openlibrary/code.py", line 931, in save_error
web-1           |     error = web.safestr(web.djangoerror())
web-1           |                         ^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/debugerror.py", line 307, in djangoerror
web-1           |     djangoerror_r = Template(djangoerror_t, filename=__file__, filter=websafe)
web-1           |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 932, in __init__
web-1           |     code = self.compile_template(text, filename)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 1003, in compile_template
web-1           |     code = Template.generate_code(
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 988, in generate_code
web-1           |     rootnode = parser.parse(text, filename)
web-1           |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 87, in parse
web-1           |     suite = self.read_suite(text)
web-1           |             ^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 175, in read_suite
web-1           |     section, text = self.read_section(text)
web-1           |                     ^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 119, in read_section
web-1           |     return self.read_block_section(text2, begin_indent)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 477, in read_block_section
web-1           |     return self.create_block_node(keyword, stmt, block, begin_indent), text
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 481, in create_block_node
web-1           |     return self.statement_nodes[keyword](stmt, block, begin_indent)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 692, in __init__
web-1           |     BlockNode.__init__(self, *a, **kw)
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 626, in __init__
web-1           |     self.suite = Parser().read_suite(block)
web-1           |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 175, in read_suite
web-1           |     section, text = self.read_section(text)
web-1           |                     ^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 119, in read_section
web-1           |     return self.read_block_section(text2, begin_indent)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 477, in read_block_section
web-1           |     return self.create_block_node(keyword, stmt, block, begin_indent), text
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 481, in create_block_node
web-1           |     return self.statement_nodes[keyword](stmt, block, begin_indent)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 626, in __init__
web-1           |     self.suite = Parser().read_suite(block)
web-1           |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 175, in read_suite
web-1           |     section, text = self.read_section(text)
web-1           |                     ^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 126, in read_section
web-1           |     return self.readline(text)
web-1           |            ^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 198, in readline
web-1           |     node, line = self.read_node(line)
web-1           |                  ^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 224, in read_node
web-1           |     return self.read_expr(text, escape=escape)
web-1           |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 362, in read_expr
web-1           |     simple_expr()
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 271, in simple_expr
web-1           |     extended_expr()
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 277, in extended_expr
web-1           |     lookahead = tokens.lookahead()
web-1           |                 ^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 338, in lookahead
web-1           |     self.items.append(self._next())
web-1           |                       ^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 343, in _next
web-1           |     return next(self.iteritems)
web-1           |            ^^^^^^^^^^^^^^^^^^^^
web-1           |   File "/home/openlibrary/.local/lib/python3.12/site-packages/web/template.py", line 318, in get_tokens
web-1           |     for t in tokenize.generate_tokens(readline):
web-1           |   File "/usr/local/lib/python3.12/tokenize.py", line 543, in _generate_tokens_from_c_tokenizer
web-1           |     raise TokenError(msg, (e.lineno, e.offset)) from None
web-1           | tokenize.TokenError: ('unterminated string literal (detected at line 1)', (1, 4))

Relevant url?

Steps to Reproduce

  1. Run the Open Library Docker development environment with an image based on Python 3.12.
  2. Visit any page on the website that loads a template and notice the only thing in the browser is Internal Server Error, and the Docker console shows an error ending with something like: tokenize.TokenError: ('unterminated string literal (detected at line 1)', (1, 4)).

Proposal & Constraints

I propose we fix the bug. Likely by making a change to web.py. But it will require more investigation to find the root of the error.

Related files

Stakeholders

@cdrini

scottbarnes commented 10 months ago

I submitted a PR over at webpy/webpy#784 to fix this.

cdrini commented 8 months ago

Scott fixed this one! 🥳

tfmorris commented 8 months ago

The problem has been patched, but I don't think there's been a new release of web.py yet. Once a release is made, the dependencies here will need to be updated with the new web.py version.

scottbarnes commented 8 months ago

I had updated requirements.txt here to point to the commit with the patch, which I would like to declare was very much a team effort with @tfmorris: https://github.com/internetarchive/openlibrary/blob/4043d671afe60105063fdcd12f59ffe36b087cbc/requirements.txt#L11

Except for any errors. Those were all mine.