Temporary fix for Turkish

sergiolaverde0 commented 3 months ago

This allows to install Turkish again from the extra languages tab, but breaks every other language as a result.

One commit also includes a fix for the breaking change in one of the dependencies.

I'm targeting main because hotfix has not been fast forwarded and I want to keep the diff readable.

simjanos-dev commented 3 months ago

Thank you so much!

The cleaner package fix is also in the dev branch, but I forgot to change it in the lines where its being used. It's weird, because I think we had this issue before and was fixed months ago.

Can you please check if you apply this fix, install Turkish, and then restart the tokenizer.py it still works? I think I had an issue, when I did what you did in this fix, and my tokenizer.py refused to start.

This fix would not affect already existing Turkish users, right?

I think if this works, I'll just merge it in, and we'll just wait for stanza. A dev found it, and I received no report about it from users.

The hotfix branch is not used anymore. I don't remember why we needed it, but we don't need it anymore, hotfix PR-s can be created to the main branch.

sergiolaverde0 commented 3 months ago

You are right, I got a NumPy error that wasn't present before when I restarted the container. I will update when I find a fix.

sergiolaverde0 commented 3 months ago

Numpy bumped to v2.0 not too long after we released v0.13 and broke some packages that didn't properly pin dependencies (not like we are in a place to judge), including Turkish. It doesn't happen on the first run when you install because numpy had already been imported, but triggered in later runs when trying to import it after everything was done. More details in the SO question in the comment I left.

I made it so that when installing Turkish it also installs the older numpy. This time I tested it and it worked after several restarts, with proper lemmatization in my sample text. This is a permanent solution since I don't think the Turkish package we use will be updated to account for this.

simjanos-dev commented 3 months ago

Thank you!

So, with this everything will work, for both existing and new users, even if other languages are installed?

sergiolaverde0 commented 3 months ago

lxml is giving trouble when importing text with the new image, but at least it seems unrelated to the Turkish affair since I can switch languages just fine and the script does not crash and burn. Installing Turkish will still break every other language.

I will still hold on this PR until I find a fix for the package update since they seem to have introduced some breaking change into the Cleaner object itself, unless you decide to tackle it yourself.

Traceback (most recent call last):
  File "/home/laravel/.local/lib/python3.10/site-packages/bottle.py", line 876, in _handle
    return route.call(**args)
  File "/home/laravel/.local/lib/python3.10/site-packages/bottle.py", line 1759, in wrapper
    rv = callback(*a, **ka)
  File "/app/tokenizer.py", line 429, in importBook
   content = loadBook(importFile, chapterSortMethod)
  File "/app/tokenizer.py", line 393, in loadBook
    epubPage = cleaner.clean_html(item.get_content()).decode('utf-8')
  File "/home/laravel/.local/lib/python3.10/site-packages/lxml_html_clean/clean.py", line 563, in clean_html
    doc = fromstring(html)
  File "/home/laravel/.local/lib/python3.10/site-packages/lxml_html_clean/clean.py", line 99, in fromstring
    return lxml_fromstring(_ascii_control_characters.sub("", string))
TypeError: cannot use a string pattern on a bytes-like object

simjanos-dev commented 3 months ago

I will still hold on this PR until I find a fix for the package update since they seem to have introduced some breaking change into the Cleaner object itself, unless you decide to tackle it yourself.

I could not understand the problem and find a fix in 40 minutes, and I would like to work on other stuff, so If it is okay, I'll wait for your fix.

sergiolaverde0 commented 3 months ago

I think I fixed it, but I don't get why it even happened. all that as supposed to happen was that one package was split in two with independent versions, but somehow that caused a function to break without a cursed workaround using RegEx.

I tested this to work with a book in Spanish and another one in Japanese.

simjanos-dev commented 3 months ago

I have an idea, but too lazy to test now. I think maybe we dont even have to change the package import and the naming, only the rest. It could just be a warning, because I already fixed it before a few months ago, and the actual problem could be the the part you changed in your latest commit.

And thank you for the fix. Will probably test it tomorrow and create a hotfix.

simjanos-dev commented 3 months ago

When I try to install Turkish, I get an unsuccessful installation error on the UI. When I try to refresh the page, it keeps loading. Python logs shows nothing. After a while, it starts working again, but these python error messages show up in the python logs, without any logs regarding the install request.

Did you also get these messages? Are these a problem?

I could not test Turkish import. Will have to do it in dev environment, but have no time.

Listening on http://0.0.0.0:8678/
Hit Ctrl-C to quit.

172.21.0.4 - - [26/Aug/2024 17:23:39] "GET /models/list HTTP/1.1" 200 2
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ca-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
da-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
de-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
el-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
en-core-web-sm 3.7.1 requires spacy<3.8.0,>=3.7.2, but you have spacy 3.4.4 which is incompatible.
es-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
fi-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
fr-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
hr-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
it-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
lt-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
mk-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
nb-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
nl-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
pl-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
pt-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
ro-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
sl-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
sv-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
xx-ent-wiki-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ca-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
da-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
de-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
el-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
en-core-web-sm 3.7.1 requires spacy<3.8.0,>=3.7.2, but you have spacy 3.4.4 which is incompatible.
es-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
fi-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
fr-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
hr-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
it-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
lt-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
mk-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
nb-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
nl-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
pl-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
pt-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
ro-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
sl-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
sv-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
xx-ent-wiki-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
172.21.0.4 - - [26/Aug/2024 17:25:11] "GET /models/list HTTP/1.1" 200 11
172.21.0.4 - - [26/Aug/2024 17:25:16] "GET /models/list HTTP/1.1" 200 11
172.21.0.4 - - [26/Aug/2024 17:25:39] "GET /models/list HTTP/1.1" 200 11
172.21.0.4 - - [26/Aug/2024 17:26:10] "GET /models/list HTTP/1.1" 200 11
172.21.0.4 - - [26/Aug/2024 17:26:19] "GET /models/list HTTP/1.1" 200 11

sergiolaverde0 commented 3 months ago

I too got the messages in the logs, which confirm what we already knew about Turkish breaking all other languages. But my install throws no errors and I'm able to switch to Turkish from the button on the pop up message.

I will try to replicate.

Edit: my own build over at ghcr.io/sergiolaverde0/linguacafe-python-service:turkish gives me no issues in my testing, does it trigger anything for you?

simjanos-dev commented 3 months ago

I'll try to run a test after work, then I'll make a hotfix anyway unless I find something really bad, it can't be worse than no Turkish.

Also, thank you so much for the fix!

simjanos-dev commented 3 months ago

I've tried again in dev environment.

My steps:

Click install Turkish
Error after 30~ seconds
Log shows nothing
If I refresh the languages page it keeps loading
After another 30~ seconds the errors/warnings show up in the logs
No sign of calling the install endpoint in the logs
After the error messages now suddenly all the list endpoints are logged
The languages page starts working again
Turkish import works
German (no install required) import works
Installing Japanese shows UI error
Same thing, nothing in logs and keeps loading
Works again, lot of warning logs, no sign of Japanese install
Surprisingly, Japanese imports also work.

If you give me an okay, I'll make a hotfix today.

The python logs:

Bottle v0.12.25 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8678/
Hit Ctrl-C to quit.

172.19.0.5 - - [28/Aug/2024 14:01:54] "GET /models/list HTTP/1.1" 200 2
 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ca-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
da-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
de-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
el-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
en-core-web-sm 3.7.1 requires spacy<3.8.0,>=3.7.2, but you have spacy 3.4.4 which is incompatible.
es-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
fi-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
fr-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
hr-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
it-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
lt-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
mk-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
nb-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
nl-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
pl-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
pt-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
ro-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
sl-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
sv-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
xx-ent-wiki-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ca-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
da-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
de-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
el-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
en-core-web-sm 3.7.1 requires spacy<3.8.0,>=3.7.2, but you have spacy 3.4.4 which is incompatible.
es-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
fi-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
fr-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
hr-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
it-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
lt-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
mk-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
nb-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
nl-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
pl-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
pt-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
ro-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
sl-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
sv-core-news-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
xx-ent-wiki-sm 3.7.0 requires spacy<3.8.0,>=3.7.0, but you have spacy 3.4.4 which is incompatible.
172.19.0.5 - - [28/Aug/2024 14:03:48] "GET /models/list HTTP/1.1" 200 11
172.19.0.5 - - [28/Aug/2024 14:03:48] "GET /models/list HTTP/1.1" 200 11
172.19.0.5 - - [28/Aug/2024 14:03:48] "GET /models/list HTTP/1.1" 200 11
172.19.0.5 - - [28/Aug/2024 14:03:48] "GET /models/list HTTP/1.1" 200 11
172.19.0.5 - - [28/Aug/2024 14:03:50] "GET /models/list HTTP/1.1" 200 11
172.19.0.5 - - [28/Aug/2024 14:06:07] "GET /models/list HTTP/1.1" 200 11
172.19.0.5 - - [28/Aug/2024 14:06:11] "GET /models/list HTTP/1.1" 200 11
172.19.0.5 - - [28/Aug/2024 14:06:44] "POST /tokenizer/import-text HTTP/1.1" 200 1866
/home/laravel/.local/lib/python3.10/site-packages/spacy/util.py:910: UserWarning: [W095] Model 'tr_core_news_md' (3.4.2) was trained with spaCy v3.4.2 and may not be 100% compatible with the current version (3.7.6). If you see errors or degraded performance, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate
  warnings.warn(warn_msg)
<frozen importlib._bootstrap>:914: ImportWarning: _ImportRedirect.find_spec() not found; falling back to find_module()
172.19.0.5 - - [28/Aug/2024 14:06:45] "POST /tokenizer HTTP/1.1" 200 22037
172.19.0.5 - - [28/Aug/2024 14:06:59] "GET /models/list HTTP/1.1" 200 11
172.19.0.5 - - [28/Aug/2024 14:07:01] "GET /models/list HTTP/1.1" 200 11
172.19.0.5 - - [28/Aug/2024 14:07:15] "POST /tokenizer/import-text HTTP/1.1" 200 48
<frozen importlib._bootstrap>:914: ImportWarning: _ImportRedirect.find_spec() not found; falling back to find_module()
172.19.0.5 - - [28/Aug/2024 14:07:18] "POST /tokenizer HTTP/1.1" 200 902
172.19.0.5 - - [28/Aug/2024 14:07:29] "GET /models/list HTTP/1.1" 200 11
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pathy 0.11.0 requires smart-open<7.0.0,>=5.2.1, but you have smart-open 7.0.4 which is incompatible.
tr-core-news-md 3.4.2 requires spacy<3.5.0,>=3.4.2, but you have spacy 3.7.6 which is incompatible.
WARNING: Target directory /var/www/html/storage/app/model/__pycache__ already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/_distutils_hack already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/numpy already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/spacy already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/packaging already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/murmurhash already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/catalogue already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/charset_normalizer already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/click-8.1.7.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/langcodes-3.4.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/requests-2.32.3.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/language_data already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/wasabi already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/langcodes already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/murmurhash-1.0.10.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/MarkupSafe-2.1.5.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/requests already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/smart_open already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/jinja2 already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/urllib3 already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/click already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/idna-3.8.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/numpy-1.26.4.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/spacy_loggers already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/charset_normalizer-3.3.2.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/spacy_legacy already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/certifi already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/spacy_loggers-1.0.5.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/tqdm already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/setuptools already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/cymem already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/preshed-3.0.9.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/typing_extensions-4.12.2.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/idna already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/thinc already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/setuptools-74.0.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/catalogue-2.0.10.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/distutils-precedence.pth already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/urllib3-2.2.2.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/tqdm-4.66.5.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/srsly-2.4.8.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/marisa_trie-1.2.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/blis-0.7.11.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/pydantic already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/blis already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/typing_extensions.py already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/srsly already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/markupsafe already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/spacy_legacy-3.0.12.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/preshed already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/cymem-2.0.8.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/confection already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/jinja2-3.1.4.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/marisa_trie.cpython-310-x86_64-linux-gnu.so already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/confection-0.1.5.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/packaging-24.1.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/certifi-2024.7.4.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/numpy.libs already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/language_data-1.2.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/typer already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/pkg_resources already exists. Specify --upgrade to force replacement.
WARNING: Target directory /var/www/html/storage/app/model/bin already exists. Specify --upgrade to force replacement.
172.19.0.5 - - [28/Aug/2024 14:08:44] "GET /models/list HTTP/1.1" 200 23
172.19.0.5 - - [28/Aug/2024 14:09:08] "GET /models/list HTTP/1.1" 200 23
172.19.0.5 - - [28/Aug/2024 14:09:11] "GET /models/list HTTP/1.1" 200 23
172.19.0.5 - - [28/Aug/2024 14:09:13] "GET /models/list HTTP/1.1" 200 23
172.19.0.5 - - [28/Aug/2024 14:09:32] "POST /tokenizer/import-text HTTP/1.1" 200 1754
<frozen importlib._bootstrap>:914: ImportWarning: _ImportRedirect.find_spec() not found; falling back to find_module()
172.19.0.5 - - [28/Aug/2024 14:09:33] "POST /tokenizer HTTP/1.1" 200 22956

simjanos-dev commented 3 months ago

Update: After restarting the containers, Japanese import stopped functioning. I forgot to check if Turkish was still working or not.

sergiolaverde0 commented 2 months ago

Languages other than Turkish breaking after container restart is "normal", or at least expected.

I want to believe that the other errors you encounter are exclusive to the dev environment (which admittedly I only use when I have to update it) but just in case, remember that if the worst happens, people can go back to the current version by adding VERSION=v0.13 to their .env files, and no database issues should happen since there have been no new migrations.

simjanos-dev commented 2 months ago

remember that if the worst happens, people can go back to the current version by adding VERSION=v0.13 to their .env files, and no database issues should happen since there have been no new migrations.

Good point. It's a bit late now, but I'll make an image tomorrow. Also, haven't got a report about Turkish from any actual users, so I think there won't be many people who are affected. If someone had it installed already before it broke, it should keep working for them.

simjanos-dev / LinguaCafe

Temporary fix for Turkish #337