Automatic plugin static files collection broken after install

wolflu05 commented 2 months ago

Please verify that this bug has NOT been raised before.

[X] I checked and didn't find a similar issue

Describe the bug*

I have broken down the issues with plugin static files into multiple smaller ones:

1. Static files path (fixed with #7763)

Details

If an `AppMixin` plugin before had a `static` folder and the django app was loaded, `invoke static` had already collected the static files in the root of the static folder so a file in `my-plugin/static/panel.js` was served by `/static/panel.js`. *(to remove clashes between files with the same name from multiple plugins, I placed my static files in `my-plugin/static/my-plugin/panel.js`, so it was served by `/static/my-plugin/panel.js`)* Now after the mentioned PR, a file in `my-plugin/static/panel.js` is served by `/static/plugins/my-plugin/panel.js`. I do not think this change is bad, it just needs to be documented as breaking. One good think would be to provide a `{% plugin_static "panel.js" %}` template tag which resolves to the particular plugin static url (`/static/plugins/my-plugin/panel.js`) and we back port that to 0.15 which resolves to `/static/panel.js`

2. Automatic static file copying and migrations are not run after activating a plugin

I can reproduce this in a fresh docker setup (but not in my dev setup running locally). But no idea why only in a production docker setup. (Sometimes only very few times it actually succeeds, but then the migrations are run two times, the first time without any error, and the second they error out, because obviously the table already exists, that the migrations should create). Maybe that's some type of race condition.

Goto AdminCenter > Plugins > Settings and Activate plugins that uses apps
Goto AdminCenter > Plugins > Install plugin and use inventree-bulk-plugin (or inventree-kicad, but that only tests with migrations)
Activate that plugin
See this error in the docker logs: inventree-worker | Plugin registry has no record of plugin 'inventree-bulk-plugin'
- See that the folder static/plugins/inventree-bulk-plugin does not exist (folder should exist)
- See that the migrantions have not run (should have run)
Run invoke update the files now exists and the migrations have run

3. When deactivating a plugin, the the static files don't get removed

When deactivating a plugin, the static files get don't removed automatically, a invoke static is needed. And even then, nested folders remain.

4. AppMixin Plugins with static files causes an error (fixed with #7763)

Details

(If 2 is solved, found with my local dev setup where I don't have the issue 2, or when running `invoke static`) 1. Install a plugin with the app mixin that provides static files like in (2.1 and 2.2) 2. Activate the plugin 3. See the following error: ``` Process-c8d4a6053e4d446fb9f81077681a7693 processing blue-snake-xray-beryllium 'plugin.staticfiles.copy_plugin_static_files' Copying static files for plugin 'inventree-bulk-plugin' Failed 'plugin.staticfiles.copy_plugin_static_files' (blue-snake-xray-beryllium) - [Errno 66] Directory not empty: '/Users/wolflu/Development/1_GITHUB/InvenTree/dev/static/inventree-bulk-plugin' : Traceback (most recent call last): File "/Users/wolflu/Development/1_GITHUB/InvenTree/dev/venv/lib/python3.11/site-packages/django_q/worker.py", line 97, in worker res = f(*task["args"], **task["kwargs"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/wolflu/Development/1_GITHUB/InvenTree/src/backend/InvenTree/plugin/staticfiles.py", line 66, in copy_plugin_static_files clear_static_dir(destination_prefix) File "/Users/wolflu/Development/1_GITHUB/InvenTree/src/backend/InvenTree/plugin/staticfiles.py", line 31, in clear_static_dir staticfiles_storage.delete(d) File "/Users/wolflu/Development/1_GITHUB/InvenTree/dev/venv/lib/python3.11/site-packages/django/core/files/storage/filesystem.py", line 156, in delete os.rmdir(name) OSError: [Errno 66] Directory not empty: '/Users/wolflu/Development/1_GITHUB/InvenTree/dev/static/inventree-bulk-plugin' ```

5. Property bug (can only sometimes reproduce)

When freshly installing a plugin (freshly, not previously installed), and then activating the plugin without restarting the server it can happen that it looks like this:

Error

``` Traceback (most recent call last): File "/root/.local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch response = handler(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/inventree/src/backend/InvenTree/plugin/api.py", line 341, in get settings_definition=settings, plugin=plugin.plugin_config() ^^^^^^^^^^^^^^^^^^^^^^ TypeError: MetaBase.plugin_config() missing 1 required positional argument: 'self' ```

Deployment Method

[X] Docker
[ ] Package
[ ] Bare metal
[ ] Other - added info in Steps to Reproduce

Version Information

# Version Information:
InvenTree-Version: 0.17.0 dev
Django Version: 4.2.15
Commit Hash: cc45c23
Commit Date: 2024-08-30

Database: postgresql
Debug-Mode: False
Deployed using Docker: True
Platform: Linux-6.6.31-linuxkit-x86_64-with
Installer: DOC

Active plugins: [{'name': 'InvenTreeBarcode', 'slug': 'inventreebarcode', 'version': '2.1.0'}, {'name': 'InvenTreeCoreNotificationsPlugin', 'slug': 'inventreecorenotificationsplugin', 'version': '1.0.0'}, {'name': 'InvenTreeCurrencyExchange', 'slug': 'inventreecurrencyexchange', 'version': '1.0.0'}, {'name': 'InvenTreeLabel', 'slug': 'inventreelabel', 'version': '1.1.0'}, {'name': 'InvenTreeLabelMachine', 'slug': 'inventreelabelmachine', 'version': '1.0.0'}, {'name': 'InvenTreeLabelSheet', 'slug': 'inventreelabelsheet', 'version': '1.0.0'}, {'name': 'DigiKeyPlugin', 'slug': 'digikeyplugin', 'version': '1.0.0'}, {'name': 'LCSCPlugin', 'slug': 'lcscplugin', 'version': '1.0.0'}, {'name': 'MouserPlugin', 'slug': 'mouserplugin', 'version': '1.0.0'}, {'name': 'TMEPlugin', 'slug': 'tmeplugin', 'version': '1.0.0'}]

Please verify if you can reproduce this bug on the demo site.

[ ] I can reproduce this bug on the demo site.

Relevant log output

No response

SchrodingersGat commented 2 months ago

@luwol03 a few different things to address here!

Static File Pathing

You say that the path for serving static files is different? This was not intentional. Can you provide examples of the differences?

Static File Copying Does Not Work

Let's work out why!

wolflu05 commented 2 months ago

Good idea to break it into different issues:

EDIT: I have added this list to the initial comment now

matmair commented 2 months ago

I think the migration part of 2 and 5 should be split out as individual issues as they do not seem connected with static content "stuff".

wolflu05 commented 2 months ago

I agree with point 5, but when looking deeper, issue 2 targets exactly the code added in the static files for plugins pr. It works in dev, but not, (reproducable) in a fresh docker prod setup. And I think if the static file not being collected automatically issue is solved, the task that is offloaded to run the migrations will be fixed too.

Can you reproduce that too in a fresh docker prod setup?

SchrodingersGat commented 2 months ago

@wolflu05 I think this has been addressed now, please let me know if this is incorrect...

wolflu05 commented 2 months ago

Only 1 and 4 have been fixed be me in one of my recent prs. 2,3 and 5 have been still not fixed from here: https://github.com/inventree/InvenTree/issues/7709#issuecomment-2244844925

SchrodingersGat commented 2 months ago

@wolflu05 good point. If you have some time to look into those? Otherwise I will try to get to it when I can :)

wolflu05 commented 2 months ago

Sorry, but unfortunately I won't have any time the next weeks.

matmair commented 1 month ago

Is this issue still happening?

wolflu05 commented 1 month ago

Is this issue still happening?

Yes, I have updated the initial comment now with the list where I have broken down the issue into smaller ones (previously this was only a comment in this discussion, so that this doesn't get lost). Issue 2 and 3 are still not solved. And issue 5 is not reproducible. I have tested with the current latest docker image (updated the version information accordingly).

wolflu05 commented 3 weeks ago

I have tested again with ec2051d, and 2,3 and 5 is still not resolved, so I'm reopening it.

Now removing the files after disabling the plugin is implemented (3), but the same error as (2) occurs when running without a restart. And I can actually reproduce 5 now again, that's a very strange one.

SchrodingersGat commented 3 weeks ago

@wolflu05 OK, let's work it out!

inventree / InvenTree