Issue when subscribing - Githubissues

jheidecker / lemmony

A better "All" browsing experience for Lemmy instances

MIT License

45 stars 3 forks source link

Issue when subscribing #2

Closed qol3m closed 1 year ago

qol3m commented 1 year ago

Issue happened after all 8000 communities were added, before subscribing, and now also happens when rerunning the script. (after enumerating all local communities (this might take a while)...)

8875/8875 https://testlemmy.lealternative.net/c/main: 200 re-fetching communities ready for subscription (this might take a while)... Traceback (most recent call last): File "/root/lemmony.py", line 64, in if actor_resp.json()['communities'] != []: File "/usr/lib/python3/dist-packages/requests/models.py", line 900, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib/python3.10/json/init.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.10/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

jheidecker commented 1 year ago

This was sort of a rush thing and I am by no means a world class coder. :) I will make this a docker container ASAP. In the mean time can you try using the same environment as me to eliminate possible conflicts there:

Please install Python 3.11.4 and confirm these package versions:

certifi==2023.5.7
charset-normalizer==3.1.0
idna==3.4
requests==2.31.0
urllib3==2.0.3

qol3m commented 1 year ago

But it worked at first? It added all 8875 communities, then broke when trying to subscribe.

However, when I re-run the script, it already breaks at the "adding communities" step, if I am reading this correctly.

In any case, I tried installing python3.11.4 with add-apt-repository ppa:deadsnakes/ppa and apt install python3.11 , then upgrading the packages with "pip install certifi --upgrade", then ran python3.11 lemmony.py . Same error.

Ubuntu 22.04, by the way. (hetzner vps) lemmy 0.18.0

jheidecker commented 1 year ago

Can you post the output for this request: curl "https://your.instance/api/v3/community/list?type_=All&limit=50&page=1"

asc6 commented 1 year ago

Hey, great tool! This still seems to be an issue with the docker container resulting in a reliable error for me. This only occurs on actually subscribing to the communities, not finding them. And even then it seems to do a small number ~12 just fine.

Running command $docker run ghcr.io/jheidecker/lemmony/lemmony lemmony-cli -l LEMMY_INSTANCE -u USER -p PASS -i lemmy.world

It gets JWT, finds the communities and then outputs this error during the subscribe function. `Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 971, in json return complexjson.loads(self.text, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/init.py", line 346, in loads return _default_decoder.decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/bin/lemmony-cli", line 8, in sys.exit(main()) ^^^^^^ File "/usr/local/lib/python3.11/site-packages/lemmony/cli.py", line 147, in main subscribe() File "/usr/local/lib/python3.11/site-packages/lemmony/cli.py", line 139, in subscribe print(str(idx) + "/" + local_community_count + " " + str(community_id) + ": " + str(sub_resp.json()['community_view']['subscribed'])) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 975, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 971, in json return complexjson.loads(self.text, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/init.py", line 346, in loads return _default_decoder.decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

jheidecker commented 1 year ago

can you share: curl "https://your.lemmy/api/v3/community/list?type_=All&limit=50&page=1&auth=jwt_token"

you can get a good token from any logged in web session. press f12 > network

asc6 commented 1 year ago

jheidecker commented 1 year ago

So frustrating. I cannot replicate this error! Do you want to give me a user credentials for that instance? The instance is the only variable I can't control so inconstancies in the response could be causing this. JSON is supposed to be JSON and the API should provide consistency. What version of lemmy is the instance and is there anything unusual about the setup? I know that's a hard question to answer. Sorry.

asc6 commented 1 year ago

Still getting the error in v0.0.3 (made sure to update the container). lowered the rate limit to 5. More than happy to create a user account for you for testing if helpful. UI: 0.18.1-rc.7 BE: 0.18.1-rc.4. Nothing unusual that I'm aware of. Selfhosting using the installation guide.

jheidecker commented 1 year ago

Email me the creds at jay.heidecker@gmail.com.

asc6 commented 1 year ago

Just sent the info over.

jheidecker commented 1 year ago

Just sent the info over.

Sorry, I am being lazy today. On my list for later this aft!

jheidecker commented 1 year ago

Good news! I replicated the error, just have to debug! Stay tuned.

asc6 commented 1 year ago

Looks like it fixed it! No issues on several test runs. Thanks!