Plugin isn't recognized by netbox

klaernie commented 2 years ago

Hi there!

After spending the entire yesterday on trying to get it to work, I'm now at a point where I consider the installation I was using as perfect, yet netbox still will not recognize the plugin. Another plugin installed the same way does work, so there must be something amiss that I cannot put my finger on or lack the debugging knowledge for.

Could you please take a look at the docker container I built:

Code: https://github.com/klaernie/netbox-docker
Image: https://github.com/klaernie/netbox-docker/pkgs/container/netbox (easiest way to inspect is docker run -ti ghcr.io/klaernie/netbox /bin/bash)

The configuration is not contained in the docker image, but injected later on kubernetes level, as I use https://github.com/bootc/netbox-chart to deploy my image.

This is the relevant part of values.yaml, that is then used to deploy the configuration to kubernetes: I only blanked out the token values, everything else is harmless to share. I could also upload a censored version of the entire file, but I think it would be way too far to hope that you'd setup an entire kubernetes cluster just to test that.

What did I test that makes me sure that the installation is as it should be?

starting netbox itself works - when a plugin cannot be found at all it would not start but rather crash with an ImportError exception.
Entering the container on kubernetes I can execute a python interpreter and tell it to run the module, where I do not get any error except the expected no module named netbox_proxbox.__main__, which is expected, since netbox-proxbox does not have one. But and compilation errors or missing dependencies would have been found by then.

How am I sure, that netbox does have the plugin loaded?

The sidebar entry is missing
Executing the migrations nothing pops up that isn't there in a stock netbox
the django admin UI doesn't list the plugin as installed.

This is what I tried inside the container to verify that everything seems okay:

kandre@mainframe(pts/11) ~/k8s/internal/netbox % kubectl -n netbox exec netbox-5f585c4995-96jmx -it netbox -- sh                                                                                            [130]:26479
/opt/netbox/netbox $ source /opt/netbox/venv/bin/activate
(venv) /opt/netbox/netbox $ /opt/netbox/launch-netbox.sh
⚙️ Applying configuration from /etc/unit/nginx-unit.json
2022/05/20 16:43:23 [warn] 33#33 Unit is running unprivileged, then it cannot use arbitrary user and group.
2022/05/20 16:43:23 [alert] 33#33 connect(7, unix:/opt/unit/unit.sock) succeed, address already in use
(venv) /opt/netbox/netbox $ ✅ Unit configuration loaded successfully
^C                                               
(venv) /opt/netbox/netbox $ ps                   
PID   USER     TIME  COMMAND                     
    1 1000      0:00 /sbin/tini -- /opt/netbox/docker-entrypoint.sh /opt/netbox/launch-netbox.sh
    7 1000      0:00 {unitd} unit: main v1.23.0 [unitd --no-daemon --control unix:/opt/unit/unit.sock --pid /opt/unit/unit.pid --log /dev/stdout --state /opt/unit/state/ --tmp /opt/unit/tmp/ --user unit --group root
   17 1000      0:00 {unitd} unit: controller
   18 1000      0:00 {unitd} unit: router
   22 1000      0:06 {unitd} unit: "netbox" application
   27 1000      0:05 {unitd} unit: "netbox" application
   28 1000      0:00 sh
   38 1000      0:00 ps
(venv) /opt/netbox/netbox $ python3 -m netbox_proxbox
🧬 loaded config '/etc/netbox/config/configuration.py'
🧬 loaded config '/etc/netbox/config/extra.py'
🧬 loaded config '/etc/netbox/config/logging.py'
🧬 loaded config '/etc/netbox/config/plugins.py'
/opt/netbox/venv/bin/python3: No module named netbox_proxbox.__main__; 'netbox_proxbox' is a package and cannot be directly executed
(venv) /opt/netbox/netbox $ python3 manage.py migrate
🧬 loaded config '/etc/netbox/config/configuration.py'
🧬 loaded config '/etc/netbox/config/extra.py'
🧬 loaded config '/etc/netbox/config/logging.py'
🧬 loaded config '/etc/netbox/config/plugins.py'
Operations to perform:
  Apply all migrations: admin, auth, circuits, contenttypes, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, wireless
Running migrations:
  No migrations to apply.
(venv) /opt/netbox/netbox $

Do you have any idea what I could try or how I could get more info out of the netbox, so that I can find out why loading the plugin doesn't happen?

Best regards, Andre

emersonfelipesp commented 2 years ago

Hi @klaernie! What Netbox version are you using?

Try using Netbox testing mode (using port 8000 in Django --insecure mode) and put the terminal logging output here so we can see what is going on.

Also, make sure to have Debug mode enabled on this Netbox development mode so that the terminal logging works correctly!

klaernie commented 2 years ago

My image is based on the official https://github.com/netbox-community/netbox-docker image, where the latest tag is v3.2.3. And the footer of my instance seconds that:

I've now run a second netbox instance in development mode, and port-forwarded it to the outside, but the log still looks fairly clean:

kandre@mainframe(pts/11) ~/k8s/internal/netbox % kubectl -n netbox exec netbox-5f585c4995-jwg29 -it netbox -- sh                                                                                            [130]:26485
/opt/netbox/netbox $ /opt/netbox/venv/bin/python /opt/netbox/netbox/manage.py runserver 0.0.0.0:9900 --insecure
🧬 loaded config '/etc/netbox/config/configuration.py'
🧬 loaded config '/etc/netbox/config/extra.py'
🧬 loaded config '/etc/netbox/config/logging.py'
🧬 loaded config '/etc/netbox/config/plugins.py'
🧬 loaded config '/etc/netbox/config/configuration.py'
🧬 loaded config '/etc/netbox/config/extra.py'
🧬 loaded config '/etc/netbox/config/logging.py'
🧬 loaded config '/etc/netbox/config/plugins.py'
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
May 21, 2022 - 15:59:42
Django version 4.0.4, using settings 'netbox.settings'
Starting development server at http://0.0.0.0:9900/
Quit the server with CONTROL-C.
[21/May/2022 16:01:48] "GET / HTTP/1.1" 200 102095
[21/May/2022 16:01:48] "GET /static/netbox-external.css?v=3.2.3 HTTP/1.1" 200 286568
[21/May/2022 16:01:49] "GET /static/netbox-dark.css?v=3.2.3 HTTP/1.1" 200 374410
[21/May/2022 16:01:49] "GET /static/netbox-light.css?v=3.2.3 HTTP/1.1" 200 232175
[21/May/2022 16:01:49] "GET /static/netbox.js?v=3.2.3 HTTP/1.1" 200 375393
[21/May/2022 16:01:49] "GET /static/debug_toolbar/css/toolbar.css HTTP/1.1" 200 11479
[21/May/2022 16:01:49] "GET /static/netbox_logo.svg HTTP/1.1" 200 4719
[21/May/2022 16:01:49] "GET /static/netbox_icon.svg HTTP/1.1" 200 835
[21/May/2022 16:01:49] "GET /static/netbox-print.css?v=3.2.3 HTTP/1.1" 200 727867
[21/May/2022 16:01:49] "GET /static/debug_toolbar/js/toolbar.js HTTP/1.1" 200 11275
[21/May/2022 16:01:49] "GET /static/materialdesignicons-webfont-KSYPMDN6.woff2?v=5.9.55 HTTP/1.1" 200 325244
[21/May/2022 16:01:49] "GET /static/debug_toolbar/css/print.css HTTP/1.1" 200 43
[21/May/2022 16:01:49] "GET /static/debug_toolbar/js/utils.js HTTP/1.1" 200 3531
[21/May/2022 16:01:49] "GET /static/netbox.ico HTTP/1.1" 200 1174
[21/May/2022 16:02:19] "GET /login/?next=/ HTTP/1.1" 200 18539
[21/May/2022 16:02:27] "POST /login/ HTTP/1.1" 302 0
[21/May/2022 16:02:29] "GET / HTTP/1.1" 200 164284
[21/May/2022 16:02:57] "GET /__debug__/render_panel/?store_id=46ba6e40b41144e88fad1ba9e8ab0bbb&panel_id=HistoryPanel HTTP/1.1" 200 7618
[21/May/2022 16:02:57] "GET /static/debug_toolbar/js/history.js HTTP/1.1" 200 1915
[21/May/2022 16:02:59] "GET /login/?next=/ HTTP/1.1" 302 0
[21/May/2022 16:03:02] "GET / HTTP/1.1" 200 163734
[21/May/2022 16:03:03] "GET /__debug__/render_panel/?store_id=89e3c240c2a74938bcdb693c9fb0ab37&panel_id=VersionsPanel HTTP/1.1" 200 2049
[21/May/2022 16:03:10] "GET /__debug__/render_panel/?store_id=89e3c240c2a74938bcdb693c9fb0ab37&panel_id=TimerPanel HTTP/1.1" 200 1388
[21/May/2022 16:03:10] "GET /static/debug_toolbar/js/timer.js HTTP/1.1" 200 3395
[21/May/2022 16:03:13] "GET /__debug__/render_panel/?store_id=89e3c240c2a74938bcdb693c9fb0ab37&panel_id=SettingsPanel HTTP/1.1" 200 50031

One thing I noticed though, the debug sidebar has a settings panel:

The one thing I see there, is that the PLUGINS_CONFIG contains my settings for netbox-proxbox, but the PLUGINS key is missing netbox_proxbox, although it is on the input side:

/opt/netbox/netbox $ grep PLUGINS: /run/config/netbox/netbox.yaml 
PLUGINS: ["netbox_topology_views","netbox_proxbox"]

Do you know of any mechanism that would cause netbox to reject a configured plugin silently?

emersonfelipesp commented 2 years ago

Do you know of any mechanism that would cause netbox to reject a configured plugin silently?

I haven't seen this yet. Could you temporarily remove netbox_topology_views configuration both from PLUGINS and PLUGINS_CONFIG variable, so that we can see if there is any change?

Also, make sure to enter Netbox virtual environment and see if netbox_proxbox shows up when listing the packages of the venv by issuing the pip3 list command. If it does not shows up in the list, try running the python3 setup develop again following Proxbox installation instructions.

Don't forget to run database migrations:

Lastly, I will take some time to see line by line of the output you just put here and try to find anything that can be leading to this error.

emersonfelipesp commented 2 years ago

I currently don't use Netbox Docker on production use or in development one. But I will create an lab using this, although I don't think there should be any different behavior from the manual process.

klaernie commented 2 years ago

Just now I found my issue!

I included a configuration.py snippet for the first plugin that I installed, in order to get manage.py's collectstatic mode to run. But it didn't work, but I never undid everything, just kept iterating and left the old code in. This meant, that I had a configuration.py snippet, that was used preferably over the main configuration.

All I needed to do was https://github.com/klaernie/netbox-docker/commit/76a99eca375b820d766fee7524edc76837314eda in order to remove that old cruft, and suddenly the plugin appeared where it should.

Now it only remains to verify my work on making footer.html be accepted as a template.

Thanks a lot, @emersonfelipesp ! I'm truely sorry I wasted some of your time on this issue.

emersonfelipesp commented 2 years ago

Now it only remains to verify my work on making footer.html be accepted as a template.

To solve it, you'll have to change settings.py by following this step of the tutorial:

1.3.2. Change Netbox 'settings.py' to include Proxbox Template directory

Probably on the next release of Netbox, it will not be necessary to make the configuration below! As the Pull Request #8733 got merged to develop branch

Edit /opt/netbox/netbox/netbox and find TEMPLATE_DIR section

### How it is configured by default (Netbox >= v3.2.0):

TEMPLATES_DIR = BASE_DIR + '/templates'
TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [TEMPLATES_DIR],
        'APP_DIRS': True,
        'OPTIONS': {
            'builtins': [
                'utilities.templatetags.builtins.filters',
                'utilities.templatetags.builtins.tags',
            ],
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.template.context_processors.media',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
                'netbox.context_processors.settings_and_registry',
            ],
        },
    },
]

### How it MUST be configured to Proxbox work:

TEMPLATES_DIR = BASE_DIR + '/templates'

# PROXBOX CUSTOM TEMPLATE
PROXBOX_TEMPLATE_DIR = BASE_DIR + '/netbox-proxbox/netbox_proxbox/templates/netbox_proxbox'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [TEMPLATES_DIR, PROXBOX_TEMPLATE_DIR],  # <--- IMPORTANT
    # The Parameters below is equal to default Netbox config                               
        'APP_DIRS': True,
        'OPTIONS': {
            'builtins': [
                'utilities.templatetags.builtins.filters',
                'utilities.templatetags.builtins.tags',
            ],
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.template.context_processors.media',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
                'netbox.context_processors.settings_and_registry',
            ],
        },
    },
]

Example of error if not configuring the code above on `settings.py`

I did it because I had to change the base/layout.html from Netbox, since there is no Jinja2 block to fill with custom information into the footer HTML tag

I will have to think about it later, on how to make netbox-proxbox installation easier. But I have some ideas to implement on netbox-plugins-store that will make all Plugin's installation easier than how it currently is.

I'm happy you solved it. If you need any other help in the future, feel free to open a Discussion or Issue about it.

klaernie commented 2 years ago

To solve it, you'll have to change settings.py by following this step of the tutorial:

I know. I already implemented it in https://github.com/klaernie/netbox-docker/blob/main/Dockerfile#L12-L13 or at least so I thought.

But now looking at it I now notice my problem: the README snipped clearly expects a full checkout of netbox-proxbox under /opt/netbox/netbox-proxbox, but when I install using the build package this of cause doesn't work, since there is no checkout at this place. So I'll either have to figure out a way to dynamically obtain the right path, or do the same as I did for the other plugin I installed: hardcode the path into the module.

I've seen that you wrote about the footer issue and the open issue in netbox - which has been merged. So shouldn't there now be a way to implement this directly?

But I have some ideas to implement on netbox-plugins-store that will make all Plugin's installation easier than how it currently is.

There is also a downside: unless the netbox-plugins-store stores which plugins to install and configure inside the database it will break horrendously in containerized environments. The deployment using the helm chart goes to probably the most extreme approach: netbox is running as a non-root user, all application source code is in the readonly mounted rootfs, and only the few places where the application is expected to write to (namely the media and reports directories) are writable at all. So running the plugin-store on a containerized deployment will break unless you pull some very clever stunts.

From my perspective I'd rather prefer a way where every plugin is installable via pip, and everything like adding new template paths is hookable by the main netbox app.

klaernie commented 2 years ago

So, turns out it's not too much involved to generate the right path: https://github.com/klaernie/netbox-docker/commit/ad1775ef1f40a562ea6cfcf890ecc8c3c1ddb24e Python does help here, by telling us the path to site-packages, and everything underneath is determined by netbox-proxbox, so that's probably the most reliable way.

klaernie commented 2 years ago

just because I think it will make you happy to see it:

emersonfelipesp commented 2 years ago

just because I think it will make you happy to see it: ![image](https://user-

It really makes me happy to see Proxbox being useful to you, and things working as expected (or almost like we expected). It is always good to have people who not only is using the project, but contributed a lot to it. Thank you!

netdevopsbr / netbox-proxbox