Some errors in the latest version

[sirzento]

I still get some errors when running on the latest version. Here is the log: https://pastebin.com/raw/hEE41wNS

So the past version did kinda work for me. At first it was throwing a few errors, restarted a few times but then it didn't crash over night. I will write an update if it will not crash after a few restarts.

Edit: After an hour it is now working. https://i.imgur.com/OI5UzXY.png

[kacperkr90]

It will crash few times after start, but it should eventually start working as intended.

Crashes are caused by selenium server which need some time to setup itself after starting. After a while it can start accepting new sessions. It is mentioned here.

I could fix it, but I think that it is minor inconvenience. I also don't want to overcomplicate existing code by adding the waiting logic.

EDIT: And don't worry, selenium server configures itself in about a minute so after this time you will start collecting your capsules 🏆 :D

[sirzento]

Ok thanks for the info! I thought it was still unstable on my pi because of all the error messages at the beginning.

[kacperkr90]

By the way, I found some free time and fixed it. Now farmer container will wait for the selenium grid container.

Important note: You have to run your config using docker compose plugin command - docker compose not docker-compose. I updated README.md. You should also check note about PASSWORD environment variable.

[sirzento]

About using docker compose and not docker-compose: https://i.imgur.com/HhzIN84.png

But it also runs and works with docker-compose

[kacperkr90]

Oh, on my OS (Raspbian 11) I couldn't run my config because depends_on with condition requires 3.9 version syntax and standalone docker-compose library installed from stable repositories was too old.

I had to install plugin to make it work. Anyway thanks for the info, I'll change README.md and add version requirement :)

[sirzento]

Got an error just now: https://pastebin.com/raw/4Y4pZdb5

btw I am also using raspbian 11

Edit: After an hour of error messages it is working again

[kacperkr90]

I managed to reproduce this error.

It started working after the hour, because exactly after the hour it reached it DRAIN count (3) and restarted. I initially set 1200 seconds for selenium session timeout which turned out to be an overkill (1200s * 3 / 60 = 1h).

I added autoheal mechanism which should improve the time to get the setup working after the failure. Additionally you should now add new env variable to api.env file. Check the README for more info.

[sirzento]

Ok I updated it. For your info: there wasn't a single error message overnight., but there was also no game played overnight.

Edit: No errors at start up. very nice.

[kacperkr90]

I think the real test will be at weekend, when multiple matches take place at once. I also think the crashes will be unavoidable, because lolesport page is kind of buggy in some places.

What we can work on is the recovery/autohealing of this setup. My goal is to make such setup which handles crashes on it's own and can auto-heal in short period of time. To the point, where you can set up it once and forget about it :D

[sirzento]

I also think the crashes will be unavoidable, because lolesport page is kind of buggy in some places.

Is that just a linux/pi thing? On windows it did work very nice with multiple matches at once. Do you know what exactly is happening? Is the browser crashing/freezing because of the low power of the pi? I have it currently running on a pi 3 without running anything else on the pi but the RAM is always near full with 300mb to 400mb SWP usage.

[kacperkr90]

Is that just a linux/pi thing? On windows it did work very nice with multiple matches at once. Do you know what exactly is happening?

It could be. I think that it can be result of combined low memory and virtualization. I recommend setting WAIT_VALUES_MULTIPLIER to bigger numer, like 3. It will allow selenium bot to wait more time and javascript can be pretty intensive on memory, i.e. some elements could need more time to render when running on containerized environment.

but the RAM is always near full with 300mb to 400mb SWP usage.

Good catch. You reminded me that Java applications will try to use all free memory if they can. Containerized JVM sees all memory of host OS and will try to utilize it. I'll add some resource quotas in free time.

[sirzento]

So now that there is a game live again, the problems are coming back. Here is my full log of today: https://pastebin.com/raw/FvwNaEei

Also strange that the bot often says that the game is not eligible for rewards.

[kacperkr90]

It's really strange, because I did not get any errors today and bot was watching multiple streams at once. AND I was testing setup with 256mb xmx for jvm.

Also strange that the bot often says that the game is not eligible for rewards.

I think it's somehow related to slower javascript rendering of DOM elements inside the container.

You can see it yourself. Open any stream at lolesports.com using Twitch as provider and refresh the page. You will see that there is a time window where rewards icon is showing that stream is not engilible for rewards. After that time period stream becomes engilible for rewards.

Now lests look at logs from my device.

As we can see the time period to render "engibility" of rewards is about 30 seconds - on my device. Look at this line in main.py. From what I see from your logs you have WAIT_VALUES_MULTIPLIER=1 which equals 15 seconds of waiting for engibility and this could not be sufficient.

Could you please, pull new changes (master) from git and update api.env to compile with api.env-templates and set WAIT_VALUES_MULTIPLIER=3 for tomorrow streams? And please, give feedback if something went wrong of course :)

[sirzento]

I was already on the latest version btw.

I did increase the wait multiplier to 3. If I still encounter new errors I will post them here.

btw I have an pretty old sd card running in the pi. Could some problems occure due to bad storage performance? Because I have 600mb used in the swp memory... Screenshot of htop: https://i.imgur.com/l45bmdc.png

Edit: So it first there where again a few error messages at the start but it seems to be working now: https://pastebin.com/raw/9YgpXHdS

Edit: It also now say this:

INFO: 2022/09/10 21:46:17 - There are 0 matches live
INFO: 2022/09/10 21:46:18 - https://lolesports.com/live/turkiye-sampiyonluk-ligi has finished
INFO: 2022/09/10 21:46:35 - https://lolesports.com/live/lcs has finished
INFO: 2022/09/10 21:46:51 - https://lolesports.com/live/lla has finished
INFO: 2022/09/10 21:47:01 - Next check: 2022-09-10 21:52:01.498298

but all matches are still online. The farmer on windows doesn't say that they endet. Very strange. After the next check it will open all 3 streams again but it takes like 20min for that..

Edit: a few hours later there where an hour of error messages again :/

[kacperkr90]

I made couple of improvement which (I hope) will reduce downtimes (like hours of error messages), cpu, memory usage and maybe reduce "Match not engilible for rewards" failure rate.

To update the setup please follow standard steps but do docker-compose down before git pull. I dropped autoheal mechanism and replaced it with graceful shutdown, so resources must be dropped before updating docker-compose.

[sirzento]

Just for your info: Everything did work now! No errors at the start of the stream or while streaming. Only strange thing is that it was saying that eu masters was finished at 17:25 but the match did go till 17:45. But idk if this is a pi problem or a generell one with the software.

Log: https://pastebin.com/raw/09WDNjp6

Update: With todays game everything worked!

[kacperkr90]

That's great! Let me know if you encounter any major issues in the future. Will will work on them. Happy farming! :D