search

Maker-Management-Platform / agent

BSD 2-Clause "Simplified" License
16 stars 2 forks source link

Not completing discovery #130

Open johnwhobbs opened 1 month ago

johnwhobbs commented 1 month ago

I have multiple projects with hundreds of .stl files. All of the files are scanned and added to the library; however the process that creates the PNG files for each one just stops at some point and will never restart, even if I start a new discovery scan or reboot the container. And to accomplish this, I had to add .stl files to the library folder a little at a time. If they were all in there at first start of the Agent container, discovery would get to some point and just stop. Files aren't added to the library and PNG files are not created.

EduardoOliveira commented 1 month ago

Hello, Could you please provide the logs of the agent container during the discovery phase? Doing that on a fresh installation, meaning an empty data folder ( not library ).

Thank you

johnwhobbs commented 1 month ago

fd5c860a6fa2ecff885ddb2ea896be93b4bedbb954c780f1f2aee3f69b05e54a-json.log

EduardoOliveira commented 1 month ago

From the logs, you have a single project with a tone of files, right? The project does show up in the UI right? Are models missing from the UI? Are some models present but without the thumbnail? Can you post a screenshot? In what hardware are you running the container? CPU RAM and architecture (arm or amd64)?

Sorry for all the questions but I need to understand the problem, from the logs, it discovers the content, looks like the assets are created, they are sent to the processing queue, but there are no errors....

johnwhobbs commented 1 month ago

There are 2 projects, one with 511 files and one with 656. Only one project shows in the UI and it does not contain all the models and most of the thumbnails are missing. I am running this on a QNAP NAS with quad core AMD64 CPU and 32GB RAM. I also tested on my personal PC which has an Intel 13900K and 64GB RAM. Same behavior.

Screenshot 2024-05-21 051620
johnwhobbs commented 1 month ago

When browsing the SQL Lite database, there are only 314 records in the project_assets table and 1 record in the projects table. This matches what is shown in the UI, but does not match what is in the library folder.

johnwhobbs commented 1 month ago

Any ideas?

EduardoOliveira commented 1 month ago

Hi, I've looked a bit at it, I seems there is a deadlock behaviour during the discovery process. Given the most recent changes I think it's related to the fronted notifications.... I haven't had a chance to try, but some combination of starting mmp and going to the Web UI, refreshing to make sure the event channel is connected, avoiding the channel full error messages.

I need to confirm my theory and make a fix...

johnwhobbs commented 1 month ago

Thanks. I would be happy to test it when you are ready

EduardoOliveira commented 1 month ago

I've found the issue. Currently, the render queue is limited to 256 pending requests and the processing queue limited to 100 requests... If your library is BIG they fill up and enter a deadlock state... As a quick and dirty fix, I'm gonna increase the size to 9999999, it should be enough... I'm currently reprocessing a 17gb library, and it's working... I'll post back when the bug fix is published

EduardoOliveira commented 1 month ago

@johnwhobbs Could you please test this package https://github.com/Maker-Management-Platform/agent/pkgs/container/agent/220710051?tag=130-not-completing-discovery ?

johnwhobbs commented 1 month ago

Works great now. All models present and thumbnails created for each. Thanks!!!!

EduardoOliveira commented 1 month ago

I'm gonna push this as an hotfix