v3.5 reports & scripts cleanup

[jeremystretch]

Proposed Changes

Starting an issue to capture miscellaneous cleanup items relating to recent work around custom scripts and reports in preparation for the v3.5 release.

Justification

This issue will serve as a standing collector for miscellanea that don't quite warrant their own individual issue. Substantial work items should still have individual issues opened.

(Comments hidden below have been resolved in the 12081-cleanup branch.)

[arthanson]

When there are no reports the message should be changed as reports are no longer placed in the directory but need to be uploaded. There should also probably be a message here for if you upgrade - Note: If you are not seeing your reports here and have upgraded NetBox...

Same thing for scripts.

[arthanson]

Somewhat of a side issue, but the docs for running RQ should probably be improved, I'd suggest adding a section to "Installation and Upgrade" for running "python netbox/manage.py rqworker", there is only one part in the documents I could find under "Plugins -> Developing Plugins -> Background Tasks" a note on "Configuring the RQ worker process" where it shows using the management command. The RQ docs just show "rq worker --with-scheduler" which is probably not what you want.

[arthanson]

I'm using the default report from: https://docs.netbox.dev/en/stable/customization/reports/ it is in the directory for 3.4 and uploaded for 3.5 with demo data the database should be pretty much the same but I am getting no results for the job run under 3.5:

[arthanson]

As reports are no longer in a directory to look at, it would be really useful to be able to look at the script itself once uploaded, I.E. on the report details page show the content of the file - otherwise you might have no idea what the script is if someone else uploaded it, or if you did a couple versions no idea which one it is:

Script run also shows no output using script from https://docs.netbox.dev/en/stable/customization/custom-scripts/ - on NetBox 3.4 it does show output.

[arthanson]

Reports detail should really have a "reports" or "script" tab like scripts to show the contents of the uploaded report script.

[arthanson]

I changed the device script and re-uploaded it and got the following error - was from having it the same filename but it should probably display a error modal instead of crash.:

Traceback (most recent call last):
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "/Users/ahanson/dev/work/netbox/netbox/netbox/views/generic/object_views.py", line 167, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/Users/ahanson/dev/work/netbox/netbox/netbox/views/generic/base.py", line 26, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/Users/ahanson/dev/work/netbox/netbox/utilities/views.py", line 99, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 142, in dispatch
    return handler(request, *args, **kwargs)
  File "/Users/ahanson/dev/work/netbox/netbox/netbox/views/generic/object_views.py", line 261, in post
    obj = form.save()
  File "/Users/ahanson/dev/work/netbox/netbox/core/forms/model_forms.py", line 111, in save
    return super().save(*args, **kwargs)
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/forms/models.py", line 548, in save
    self.instance.save()
  File "/Users/ahanson/dev/work/netbox/netbox/extras/models/reports.py", line 68, in save
    return super().save(*args, **kwargs)
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/db/models/base.py", line 812, in save
    self.save_base(
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/db/models/base.py", line 863, in save_base
    updated = self._save_table(
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/db/models/base.py", line 1006, in _save_table
    results = self._do_insert(
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/db/models/base.py", line 1047, in _do_insert
    return manager._insert(
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1791, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1660, in execute_sql
    cursor.execute(sql, params)
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/debug_toolbar/panels/sql/tracking.py", line 230, in execute
    return self._record(self.cursor.execute, sql, params)
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/debug_toolbar/panels/sql/tracking.py", line 154, in _record
    return method(sql, params)
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 102, in execute
    return super().execute(sql, params)
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/Users/ahanson/dev/work/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "core_managedfile_unique_root_path"
DETAIL:  Key (file_root, file_path)=(reports, devices.py) already exists.

[sleepinggenius2]

If reports/scripts are no longer going to be in a directory, how do we handle supporting files moving forward, Python or non-Python? For example, today we use a YAML configuration file that gets read by different reports/scripts, so that we can use the same reports/scripts in different environments (dev, staging, prod, etc.), but abstract out any variables that are specific to the particular environment. We also have some Python library files with shared code that is used by multiple reports/scripts. All of these files live within REPORTS_ROOT and SCRIPTS_ROOT today.

[jeremystretch]

When creating a report/script from a DataFile, the path of the origin file is not replicated. For example, a DataFile with a path of foo/bar/myreport.py will create a file myreport.py in the reports root.

[jeremystretch]

@sleepinggenius2 this issue is only for recording observed bugs & limitations in ongoing work. Please hold such discussion until the beta release has been made available for testing.

[jeremystretch]

Attempting to render the reports list fails if a single report triggers an unhandled exception (such as an ImportError).

[sleepinggenius2]

@sleepinggenius2 this issue is only for recording observed bugs & limitations in ongoing work. Please hold such discussion until the beta release has been made available for testing.

I apologize. I read through #12059 and it wasn't clear where this was being accounted for, so I thought it was a relevant question to ask in this context with respect to a potential limitation. I will wait for the beta release for further comments.

[jeremystretch]

@kkthxbye-code has provided an example repo with a few scripts to help test upgrading & loading. There are currently two issues:

1. Relative imports from within a module don't work. We likely just need to tweak the module loading logic.
2. Empty modules get replicated as database objects. This isn't necessarily a problem, however it can be confusing. We'll need to figure out the ideal strategy for managing these files.