netbox-community / netbox

The premier source of truth powering network automation. Open source under Apache 2. Public demo: https://demo.netbox.dev
http://netboxlabs.com/oss/netbox/
Apache License 2.0
15.39k stars 2.49k forks source link

Lots of page not found errors while all pages exists after enabling Sentry #15085

Closed dylode closed 4 months ago

dylode commented 5 months ago

Deployment Type

Self-hosted

NetBox Version

v3.7.1

Python Version

3.10

Steps to Reproduce

  1. Enable Sentry on the Netbox instance
  2. Create an IP address, device or custom field (so far we only noticed these models having this issue).
  3. Try to use the API to retrieve it:
curl -X GET -L  \
-H "Authorization: Token <Token>" \
https://netbox.example.net/api/extras/custom-fields/<ID here>
  1. Notice how Sentry adds a "Page not found" error log for this call while you got an 200 OK response.

Expected Behavior

Since the response is a 200 OK, it should not report to Sentry that the page is not found.

Observed Behavior

We are getting thousands of "page not found" errors in Sentry of our Netbox for pages (all related to API calls) that do exists just fine.

Here is a JSON dump of one of these errors in Sentry:

{"event_id":"447c3796541241f2a4bfb5bc0606f394","project":26,"release":"3.7.1","dist":null,"platform":"python","message":"Page not found","datetime":"2024-02-08T13:28:26+00:00","tags":[["environment","production"],["level","error"],["runtime","CPython 3.10.12"],["runtime.name","CPython"],["release","3.7.1"],["user","ip:::ffff:x.x.x.x"],["server_name","netbox-live-371"],["transaction","/api/dcim/devices/5710"],["url","https://netbox.redacted.net/api/dcim/devices/5710"]],"_meta":{"request":{"headers":{"2":{"1":{"":{"rem":[["@password:filter","s",0,10]],"len":46}}}}}},"_metrics":{"bytes.ingested.event":5495,"bytes.stored.event":6515},"breadcrumbs":{"values":[{"timestamp":1707398906.294575,"type":"redis","category":"redis","level":"info","message":"GET ':1:config'","data":{"db.operation":"GET","redis.command":"GET","redis.is_cluster":false,"redis.key":":1:config"}},{"timestamp":1707398906.294856,"type":"redis","category":"redis","level":"info","message":"GET ':1:config_version'","data":{"db.operation":"GET","redis.command":"GET","redis.is_cluster":false,"redis.key":":1:config_version"}},{"timestamp":1707398906.295473,"type":"default","category":"query","level":"info","message":"SELECT \"core_configrevision\".\"id\", \"core_configrevision\".\"created\", \"core_configrevision\".\"comment\", \"core_configrevision\".\"data\" FROM \"core_configrevision\" ORDER BY \"core_configrevision\".\"created\" ASC LIMIT 1"}]},"contexts":{"runtime":{"name":"CPython","version":"3.10.12","build":"3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]","type":"runtime"},"trace":{"trace_id":"52bf9c69cc9c43d7b0e395a84d4e14f5","span_id":"a5ef9987469bb720","parent_span_id":"b8a150a1f93a8a22","op":"middleware.django","status":"unknown","client_sample_rate":1.0,"description":"django_prometheus.middleware.PrometheusAfterMiddleware.__call__","type":"trace"}},"culprit":"/api/dcim/devices/5710","environment":"production","extra":{"sys.argv":["/opt/netbox/venv/bin/gunicorn","--pid","/var/tmp/netbox.pid","--pythonpath","/opt/netbox/netbox","--config","/opt/netbox/gunicorn.py","netbox.wsgi"]},"fingerprint":["{{ default }}"],"grouping_config":{"enhancements":"eJybzDRxc15qeXFJZU6qlZGBkbGugaGuoeEEAHJMCAM","id":"newstyle:2023-01-11"},"hashes":["d0fbda9855d118740f1105334305c126"],"key_id":"31","level":"error","logentry":{"formatted":"Page not found"},"logger":"","metadata":{"title":"Page not found"},"modules":{"aniso8601":"9.0.1","asgiref":"3.7.2","async-timeout":"4.0.3","attrs":"23.2.0","babel":"2.14.0","bcrypt":"4.1.2","bleach":"6.1.0","certifi":"2023.11.17","cffi":"1.16.0","charset-normalizer":"3.3.2","click":"8.1.7","colorama":"0.4.6","cryptography":"42.0.1","defusedxml":"0.8.0rc2","django":"4.2.9","django-cors-headers":"4.3.1","django-debug-toolbar":"4.2.0","django-filter":"23.5","django-graphiql-debug-toolbar":"0.2.0","django-js-asset":"2.2.0","django-mptt":"0.14.0","django-pglocks":"1.0.4","django-prometheus":"2.3.1","django-redis":"5.4.0","django-rich":"1.8.0","django-rq":"2.10.1","django-tables2":"2.7.0","django-taggit":"5.0.1","django-timezone-field":"6.1.0","djangorestframework":"3.14.0","drf-spectacular":"0.27.0","drf-spectacular-sidecar":"2024.1.1","feedparser":"6.0.11","future":"0.18.3","ghp-import":"2.1.0","graphene":"3.3","graphene-django":"3.0.0","graphql-core":"3.2.3","graphql-relay":"3.2.0","gunicorn":"20.1.0","idna":"3.6","inflection":"0.5.1","jinja2":"3.1.3","jsonschema":"4.21.1","jsonschema-specifications":"2023.12.1","junos-eznc":"2.7.0","lxml":"5.1.0","markdown":"3.5.2","markdown-it-py":"3.0.0","markupsafe":"2.1.4","mdurl":"0.1.2","mergedeep":"1.3.4","mkdocs":"1.5.3","mkdocs-autorefs":"0.5.0","mkdocs-material":"9.5.4","mkdocs-material-extensions":"1.3.1","mkdocstrings":"0.24.0","mkdocstrings-python-legacy":"0.2.3","napalm":"4.1.0","napalm-ce":"0.2.0","napalm-huawei-vrp":"1.1.0","ncclient":"0.6.15","netaddr":"0.10.1","netmiko":"4.3.0","netutils":"1.6.0","ntc_templates":"4.2.0","oauthlib":"3.2.2","packaging":"23.2","paginate":"0.5.6","paramiko":"3.4.0","pathspec":"0.12.1","pillow":"10.2.0","pip":"23.3.2","platformdirs":"4.1.0","prometheus-client":"0.19.0","promise":"2.3","psycopg":"3.1.17","psycopg-binary":"3.1.17","psycopg-pool":"3.2.1","pycparser":"2.21","pyeapi":"1.0.2","pygments":"2.17.2","pyjwt":"2.8.0","pymdown-extensions":"10.7","pynacl":"1.5.0","pyparsing":"3.1.1","pyserial":"3.5","python-dateutil":"2.8.2","python3-openid":"3.2.0","pytkdocs":"0.16.1","pytz":"2023.4","pyyaml":"6.0.1","pyyaml_env_tag":"0.1","redis":"5.0.1","referencing":"0.33.0","regex":"2023.12.25","requests":"2.31.0","requests-oauthlib":"1.3.1","rich":"13.7.0","rpds-py":"0.17.1","rq":"1.15.1","scp":"0.14.5","sentry-sdk":"1.40.2","setuptools":"59.6.0","sgmllib3k":"1.0.0","six":"1.16.0","social-auth-app-django":"5.4.0","social-authr-core":"4.5.1","sqlparse":"0.4.4","svgwrite":"1.4.3","tablib":"3.5.0","text-unidecode":"1.3","textfsm":"1.1.3","transitions":"0.9.0","ttp":"0.9.5","ttp-templates":"0.3.6","typing_extensions":"4.9.0","tzdata":"2023.4","uritemplate":"4.1.1","urllib3":"2.1.0","watchdog":"3.0.0","webencodings":"0.5.1","wheel":"0.42.0","yamlordereddictloader":"0.4.2"},"nodestore_insert":1707398984.177547,"received":1707398906.31036,"request":{"url":"https://netbox.redacted.net/api/dcim/devices/5710","method":"GET","headers":[["Accept","application/json"],["Accept-Encoding","gzip"],["Authorization","[Filtered]"],["Baggage","sentry-trace_id=52bf9c69cc9c43d7b0e395a84d4e14f5,sentry-sample_rate=1,sentry-transaction=GET%20app_redacted_supportrequest_adminsupportrequest_index,sentry-public_key=8f0f78b02f3e4b86a86ac51f1db040ae,sentry-release=dev-master%407866a59,sentry-environment=prod,sentry-sampled=true"],["Connection","close"],["Host","127.0.0.1:8001"],["Sentry-Trace","52bf9c69cc9c43d7b0e395a84d4e14f5-9fa4a9b996084d92-1"],["User-Agent","Symfony HttpClient/Curl"],["X-Forwarded-Host","netbox.redacted.net"],["X-Forwarded-Proto","https"],["X-Real-Ip","::ffff:x.x.x.x"]],"env":{"REMOTE_ADDR":"127.0.0.1","SERVER_NAME":"127.0.0.1","SERVER_PORT":"8001"}},"sdk":{"name":"sentry.python.django","version":"1.40.2","integrations":["argv","atexit","dedupe","django","excepthook","logging","modules","redis","rq","stdlib","threading"],"packages":[{"name":"pypi:sentry-sdk","version":"1.40.2"}]},"timestamp":1707398906.297148,"title":"Page not found","transaction":"/api/dcim/devices/5710","transaction_info":{"source":"url"},"type":"default","user":{"ip_address":"::ffff:x.x.x.x"},"version":"7","location":null}

markkuleinio commented 5 months ago

In your examples you are missing the trailing slash, both in your curl call and in the JSON dump. Can you retest with the correct call?

$ curl -H "Authorization: Token 176d4c04ccc8f2a549ea6fd393567d9da5a796ff" \
http://netbox-test.lein.io/api/extras/custom-fields/11 -i
HTTP/1.1 301 Moved Permanently
...

$ curl -H "Authorization: Token 176d4c04ccc8f2a549ea6fd393567d9da5a796ff" \
http://netbox-test.lein.io/api/extras/custom-fields/11/ -i
HTTP/1.1 200 OK
...
<JSON output from NetBox API>
dylode commented 4 months ago

In your examples you are missing the trailing slash, both in your curl call and in the JSON dump. Can you retest with the correct call?

$ curl -H "Authorization: Token 176d4c04ccc8f2a549ea6fd393567d9da5a796ff" \
http://netbox-test.lein.io/api/extras/custom-fields/11 -i
HTTP/1.1 301 Moved Permanently
...

$ curl -H "Authorization: Token 176d4c04ccc8f2a549ea6fd393567d9da5a796ff" \
http://netbox-test.lein.io/api/extras/custom-fields/11/ -i
HTTP/1.1 200 OK
...
<JSON output from NetBox API>

Ah I see, so without the trailing slash you first get a 301 and then a 200. This 301 is then reported as a 404 in Sentry. I do wonder if this should be the case?

dylode commented 4 months ago

I have now added trailing slashes to all our API calls and the problem is fixed. Thanks markkuleinio