Server Stopps Accepting request after some time.

[Crushedice]

Greetings, My current Setup for Exceptionless is a Self-hosted Docker instance on a Vpn, together with a cloud flare Tunnel to use a Domain name owned by me.

After the initial setup for a Winforms client, it works all as intended, Error reporting, additional data, even sessions.

But after a while, the server stops accepting requests altogether, when i inspected the requests with "Fiddler", I get "HTTP/1.1 401 Unauthorized" responses. The requests from the client are the same, the whole time. I also tried both; using the Domain name , aswell as the IP+port. Same result.

Some help would be appreciated .

[niemyjski]

I'm really not sure, there is nothing we do that should cause this behavior. You'd need to dig into the application logs as well as the logs for your VPN. This is a bit outside the scope of this proj but if you see anything in the application logs of failures, please include them in this issue.

[Crushedice]

Ok, I think I found a hint. I tried a bunch of things, even the Api tests at "../docs/index.html", of which ONLY one thing worked at all: Creating a new api key, so I already thought it probably has something todo with a rate limit.

And actually I saw this, showing up in the response headers , after creating a new key:

HTTP/1.1 202 Accepted Content-Length: 0 Connection: close Date: Wed, 26 Oct 2022 21:11:56 GMT Content-Security-Policy: default-src 'self' https://js.stripe.com http://js.stripe.com;script-src 'self' https://js.stripe.com http://js.stripe.com https://cdn.jsdelivr.net http://cdn.jsdelivr.net 'unsafe-eval' 'unsafe-inline';style-src 'self' https://fonts.googleapis.com http://fonts.googleapis.com https://cdn.jsdelivr.net http://cdn.jsdelivr.net 'unsafe-inline';font-src 'self' https://fonts.gstatic.com http://fonts.gstatic.com https://cdn.jsdelivr.net http://cdn.jsdelivr.net;img-src 'self' data: https://q.stripe.com http://q.stripe.com https://www.gravatar.com http://www.gravatar.com Strict-Transport-Security: max-age=31536000; includeSubDomains Referrer-Policy: strict-origin-when-cross-origin X-Content-Type-Options: nosniff X-Frame-Options: DENY X-XSS-Protection: 1; mode=block X-Exceptionless-ConfigVersion: 4 X-RateLimit-Limit: 3500 X-RateLimit-Remaining: 3498

The X-RateLimit , did not show up on other - unauthorized 401 - attempts.

is this actually a Api-Key Rate limit? and if, how to disable it "easily" ... I'm not too keen on the whole docker thing, for me its kind of unreasonably complicated. Even though I'm a C# Developer, never used docker that much at all.

[niemyjski]

The Api Throttle limit is set to 3500 by default and if you are over the limit the Status429TooManyRequests status code will be returned. If you want to change the value, you need to set an env variable like -e EX_ApiThrottleLimit=-1

options.ApiThrottleLimit = config.GetValue(nameof(options.ApiThrottleLimit), options.AppMode == AppMode.Development ? Int32.MaxValue : 3500).NormalizeValue();

I don't think this explains the 401.

[Crushedice]

Something that got my attention was this :

namespace Exceptionless.Web.Controllers;

[Produces("application/json")]
[ApiController]
public abstract class ExceptionlessApiController : Controller {
    public const string API_PREFIX = "api/v2";
    protected const int DEFAULT_LIMIT = 10;
    protected const int MAXIMUM_LIMIT = 100;
    protected const int MAXIMUM_SKIP = 1000;

specially because, i got 3 api keys out for my users, and got ~300 reports, each, stopped working at around 100. What exactly does this max limit / how could i set the limit to a different value?

[ejsmith]

That is for paging data returned from the UI.

Can you check to see if your API keys have been disabled?

[Crushedice]

as far as i can see, they are not ScreenShot

if it helps, here's the info for the container its running in :

{
    "AppArmorProfile": "unconfined",
    "Args": [],
    "Config": {
        "AttachStderr": true,
        "AttachStdin": true,
        "AttachStdout": true,
        "Cmd": null,
        "Domainname": "rusticaland.net",
        "Entrypoint": [
            "/app/docker-entrypoint.sh"
        ],
        "Env": [
            "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "ELASTIC_CONTAINER=true",
            "discovery.type=single-node",
            "xpack.security.enabled=false",
            "ES_JAVA_OPTS=-Xms2g -Xmx2g",
            "ASPNETCORE_URLS=http://+:80",
            "DOTNET_RUNNING_IN_CONTAINER=true",
            "EX_ConnectionStrings__Storage=provider=folder;path=/app/storage",
            "EX_ConnectionStrings__Elasticsearch=server=http://localhost:9200",
            "EX_RunJobsInProcess=true",
            "EX_Html5Mode=true"
        ],
        "ExposedPorts": {
            "80/tcp": {},
            "9200/tcp": {},
            "9300/tcp": {}
        },
        "Hostname": "Sentry",
        "Image": "exceptionless/exceptionless:latest",
        "Labels": {
            "org.label-schema.build-date": "2022-10-04T10:35:41.162162476Z",
            "org.label-schema.license": "Elastic-License-2.0",
            "org.label-schema.name": "Elasticsearch",
            "org.label-schema.schema-version": "1.0",
            "org.label-schema.url": "https://www.elastic.co/products/elasticsearch",
            "org.label-schema.usage": "https://www.elastic.co/guide/en/elasticsearch/reference/index.html",
            "org.label-schema.vcs-ref": "42f05b9372a9a4a470db3b52817899b99a76ee73",
            "org.label-schema.vcs-url": "https://github.com/elastic/elasticsearch",
            "org.label-schema.vendor": "Elastic",
            "org.label-schema.version": "8.4.3",
            "org.opencontainers.image.created": "2022-10-04T10:35:41.162162476Z",
            "org.opencontainers.image.documentation": "https://www.elastic.co/guide/en/elasticsearch/reference/index.html",
            "org.opencontainers.image.licenses": "Elastic-License-2.0",
            "org.opencontainers.image.revision": "42f05b9372a9a4a470db3b52817899b99a76ee73",
            "org.opencontainers.image.source": "https://github.com/elastic/elasticsearch",
            "org.opencontainers.image.title": "Elasticsearch",
            "org.opencontainers.image.url": "https://www.elastic.co/products/elasticsearch",
            "org.opencontainers.image.vendor": "Elastic",
            "org.opencontainers.image.version": "8.4.3"
        },
        "OnBuild": null,
        "OpenStdin": true,
        "StdinOnce": true,
        "Tty": true,
        "User": "elasticsearch",
        "Volumes": {
            "/usr/share/elasticsearch/data": {}
        },
        "WorkingDir": "/app"
    },
    "Created": "2022-10-26T20:23:07.24920675Z",
    "Driver": "overlay2",
    "ExecIDs": null,
    "GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/docker/overlay2/52f93a9c656737adabf174144fd538cca88e9b010e396ecbba7e670503eed303-init/diff:/var/lib/docker/overlay2/f8c80f9cf64c86ab78b71f25d55606e5e8a7cc4b76d62b1abbbd57e2fa0cab76/diff:/var/lib/docker/overlay2/8358f929091d71bbce79f72a6c6304fc10239ec2ae80e99672ff5e3202450079/diff:/var/lib/docker/overlay2/d02346b58c3ec27dfec3c1c45b1512da9fa4b6eb3a7d26588828da76eabb3c4a/diff:/var/lib/docker/overlay2/e723bd0b2ab915c897f85a44f31aa0ccd5bc4ded6604ab50b17b7d0c8b6a0314/diff:/var/lib/docker/overlay2/a87b47ff7ae6753f4f20fa7c32c5463a55bcabf789c395fa7f5d578b54152594/diff:/var/lib/docker/overlay2/5c097086b07d8ab0ef9a0e78b9fd7052c12ddae5c61d9a0f0e49749477cb7ed6/diff:/var/lib/docker/overlay2/1187fc5259b2c78543de23d7cb7927ff88b72db2be17c2c26b68cb6d58b6553c/diff:/var/lib/docker/overlay2/319da793ac5e07ed85a7f10b97d91359c33a718369768641a9a8932dbda6a46f/diff:/var/lib/docker/overlay2/fb0d2bbf24e46ea8b99b496307f2b3d078f7edb6d1d09b23307d822bf3327a2d/diff:/var/lib/docker/overlay2/540ccf9c57a250ca44f38f129db96603755175e14d3a1a8bbe4c905430e6bbf6/diff:/var/lib/docker/overlay2/2a8731cda096912b1ba989838a1e5df2fb0db56cc5a63393611d15d2438cf9e7/diff:/var/lib/docker/overlay2/8af548bcda45186555d906cce91726cf109442adf106d40b58e3d9ba2310f1d2/diff:/var/lib/docker/overlay2/4a380ee463a9f69dd93b7af2063dcf450b41e197b3010973675f0977375f22be/diff:/var/lib/docker/overlay2/fd98653abd8c9273c505e9a0f8cd76ce3cf59df7b6f074bab295399d3888d2c4/diff:/var/lib/docker/overlay2/a750bc59989e95884d0c957a43b299341fb3439beab3d36538ba5ba2f7b35b26/diff:/var/lib/docker/overlay2/ba7f53d2a90bc7c887c3e9a10edbaef47c4a49159e75ae7e47ddd29b1104c37a/diff:/var/lib/docker/overlay2/840da7dca64695dc1babad3d92b04ea7d9ac1cd4fdd18d91489d4c79d63c1790/diff:/var/lib/docker/overlay2/f387cede87a39930a3d26ecacd1b5d9c2c6adaaf757e79e24c06ca994d07c020/diff:/var/lib/docker/overlay2/b82e10090d67cf5e8cc0916c4e3ed6d4dae21a582c35941f109ae042353f8505/diff:/var/lib/docker/overlay2/bb7060d1da0082b2d431d8615b81556e457e8e2b3ff2e3535aeea4e13e054c90/diff:/var/lib/docker/overlay2/115b5cdaa5c7284410dc9ccccc208da4c755f1fa62da97e2ec08686e9d2493e5/diff:/var/lib/docker/overlay2/844014717ae302fc47b75b7d009eaa00b102efcf69a2ad79f9257f66887ca917/diff",
            "MergedDir": "/var/lib/docker/overlay2/52f93a9c656737adabf174144fd538cca88e9b010e396ecbba7e670503eed303/merged",
            "UpperDir": "/var/lib/docker/overlay2/52f93a9c656737adabf174144fd538cca88e9b010e396ecbba7e670503eed303/diff",
            "WorkDir": "/var/lib/docker/overlay2/52f93a9c656737adabf174144fd538cca88e9b010e396ecbba7e670503eed303/work"
        },
        "Name": "overlay2"
    },
    "HostConfig": {
        "AutoRemove": false,
        "Binds": [
            "/root/esdata:/usr/share/elasticsearch/data"
        ],
        "BlkioDeviceReadBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceWriteIOps": null,
        "BlkioWeight": 0,
        "BlkioWeightDevice": [],
        "CapAdd": [
            "AUDIT_WRITE",
            "CHOWN",
            "DAC_OVERRIDE",
            "FOWNER",
            "FSETID",
            "KILL",
            "MKNOD",
            "NET_BIND_SERVICE",
            "NET_RAW",
            "SETFCAP",
            "SETGID",
            "SETPCAP",
            "SETUID",
            "SYS_CHROOT"
        ],
        "CapDrop": [
            "AUDIT_CONTROL",
            "BLOCK_SUSPEND",
            "DAC_READ_SEARCH",
            "IPC_LOCK",
            "IPC_OWNER",
            "LEASE",
            "LINUX_IMMUTABLE",
            "MAC_ADMIN",
            "MAC_OVERRIDE",
            "NET_ADMIN",
            "NET_BROADCAST",
            "SYSLOG",
            "SYS_ADMIN",
            "SYS_BOOT",
            "SYS_MODULE",
            "SYS_NICE",
            "SYS_PACCT",
            "SYS_PTRACE",
            "SYS_RAWIO",
            "SYS_RESOURCE",
            "SYS_TIME",
            "SYS_TTY_CONFIG",
            "WAKE_ALARM"
        ],
        "Cgroup": "",
        "CgroupParent": "",
        "CgroupnsMode": "private",
        "ConsoleSize": [
            0,
            0
        ],
        "ContainerIDFile": "",
        "CpuCount": 0,
        "CpuPercent": 0,
        "CpuPeriod": 0,
        "CpuQuota": 0,
        "CpuRealtimePeriod": 0,
        "CpuRealtimeRuntime": 0,
        "CpuShares": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "DeviceCgroupRules": null,
        "DeviceRequests": null,
        "Devices": [],
        "Dns": [],
        "DnsOptions": [],
        "DnsSearch": [],
        "ExtraHosts": [],
        "GroupAdd": null,
        "IOMaximumBandwidth": 0,
        "IOMaximumIOps": 0,
        "Init": true,
        "IpcMode": "private",
        "Isolation": "",
        "KernelMemory": 0,
        "KernelMemoryTCP": 0,
        "Links": null,
        "LogConfig": {
            "Config": {},
            "Type": "local"
        },
        "MaskedPaths": [
            "/proc/asound",
            "/proc/acpi",
            "/proc/kcore",
            "/proc/keys",
            "/proc/latency_stats",
            "/proc/timer_list",
            "/proc/timer_stats",
            "/proc/sched_debug",
            "/proc/scsi",
            "/sys/firmware"
        ],
        "Memory": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "MemorySwappiness": null,
        "NanoCpus": 0,
        "NetworkMode": "bridge",
        "OomKillDisable": null,
        "OomScoreAdj": 0,
        "PidMode": "",
        "PidsLimit": null,
        "PortBindings": {
            "80/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "5000"
                }
            ]
        },
        "Privileged": true,
        "PublishAllPorts": false,
        "ReadonlyPaths": [
            "/proc/bus",
            "/proc/fs",
            "/proc/irq",
            "/proc/sys",
            "/proc/sysrq-trigger"
        ],
        "ReadonlyRootfs": false,
        "RestartPolicy": {
            "MaximumRetryCount": 0,
            "Name": "no"
        },
        "Runtime": "runc",
        "SecurityOpt": [
            "label=disable"
        ],
        "ShmSize": 67108864,
        "UTSMode": "",
        "Ulimits": null,
        "UsernsMode": "",
        "VolumeDriver": "",
        "VolumesFrom": null
    },
    "HostnamePath": "/var/lib/docker/containers/2ba42ece533a7bae19769e08bd36253b8aa057db0c8cde13d7a99eb3f6e1c2b3/hostname",
    "HostsPath": "/var/lib/docker/containers/2ba42ece533a7bae19769e08bd36253b8aa057db0c8cde13d7a99eb3f6e1c2b3/hosts",
    "Id": "2ba42ece533a7bae19769e08bd36253b8aa057db0c8cde13d7a99eb3f6e1c2b3",
    "Image": "sha256:3a780c2d268748b29b80a555e557f6dda59aebc96d8cda0a1483ec78bab2328e",
    "LogPath": "",
    "MountLabel": "",
    "Mounts": [
        {
            "Destination": "/usr/share/elasticsearch/data",
            "Mode": "",
            "Propagation": "rprivate",
            "RW": true,
            "Source": "/root/esdata",
            "Type": "bind"
        }
    ],
    "Name": "/exct",
    "NetworkSettings": {
        "Bridge": "",
        "EndpointID": "a8d3789a0c67d63540033e6bced5adf7294f2ec834e41c79b41964f7a7ad56d6",
        "Gateway": "172.17.0.1",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "HairpinMode": false,
        "IPAddress": "172.17.0.3",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "MacAddress": "02:42:ac:11:00:03",
        "Networks": {
            "bridge": {
                "Aliases": null,
                "DriverOpts": null,
                "EndpointID": "a8d3789a0c67d63540033e6bced5adf7294f2ec834e41c79b41964f7a7ad56d6",
                "Gateway": "172.17.0.1",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAMConfig": {},
                "IPAddress": "172.17.0.3",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "Links": null,
                "MacAddress": "02:42:ac:11:00:03",
                "NetworkID": "9b389ebbd0a3317ec50f43df90a6055ee8a172b0ace7d6f17f5bbe4ac588eed7"
            }
        },
        "Ports": {
            "80/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "5000"
                },
                {
                    "HostIp": "::",
                    "HostPort": "5000"
                }
            ],
            "9200/tcp": null,
            "9300/tcp": null
        },
        "SandboxID": "d8630443845d3932338015632cc9a91ef6e0a1d289aa6f72e1b9d6b8fe13e7df",
        "SandboxKey": "/var/run/docker/netns/d8630443845d",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null
    },
    "Path": "/app/docker-entrypoint.sh",
    "Platform": "linux",
    "Portainer": {
        "ResourceControl": {
            "Id": 21,
            "ResourceId": "2ba42ece533a7bae19769e08bd36253b8aa057db0c8cde13d7a99eb3f6e1c2b3",
            "SubResourceIds": [],
            "Type": 1,
            "UserAccesses": [],
            "TeamAccesses": [],
            "Public": false,
            "AdministratorsOnly": true,
            "System": false
        }
    },
    "ProcessLabel": "",
    "ResolvConfPath": "/var/lib/docker/containers/2ba42ece533a7bae19769e08bd36253b8aa057db0c8cde13d7a99eb3f6e1c2b3/resolv.conf",
    "RestartCount": 0,
    "State": {
        "Dead": false,
        "Error": "",
        "ExitCode": 0,
        "FinishedAt": "2022-10-26T20:33:00.063826617Z",
        "OOMKilled": false,
        "Paused": false,
        "Pid": 99115,
        "Restarting": false,
        "Running": true,
        "StartedAt": "2022-10-26T20:33:00.717044399Z",
        "Status": "running"
    }
}

[niemyjski]

Can you check in Elasticsearch specifically the tokens index. Can you see if it's suspended or disabled.

[niemyjski]

We will add a new status to the ui to show if it's suspended.

[Crushedice]

After some bit of trying around with the API calls, i finally got some useful response that shows at least what's wrong now.

[
  {
    "id": "-----------------------------------------------",
    "organization_id": "635c256d8c84b3001861afbb",
    "project_id": "635c256d8c84b3001861afbc",
    "scopes": [
      "client"
    ],
    "is_disabled": false,
    "is_suspended": true,
    "created_utc": "2022-10-28T18:54:38.0639116Z",
    "updated_utc": "2022-10-28T18:54:38.0643681Z"
  },
  {
    "id": "-------------------------------------------",
    "organization_id": "635c256d8c84b3001861afbb",
    "project_id": "635c256d8c84b3001861afbc",
    "scopes": [
      "client"
    ],
    "is_disabled": false,
    "is_suspended": true,
    "created_utc": "2022-10-28T19:06:29.375407Z",
    "updated_utc": "2022-10-28T19:06:29.3754546Z"
  }
]

So, ya. They seem to be suspended... Any way to revert them, and prevent this from happening?

[ejsmith]

Hmm... shouldn't be happening on self hosted installs. Can you check the org api call to see if it's suspended as well?

[niemyjski]

We're going to make changes to make this more visible in the UI. @ejsmith Perhaps we should log a reason on the org with why it's suspended.

[Crushedice]

I've reinstalled my vps, as I was running into some Issues, unrelated to this. nothing much has improved, however I've managed to setup a separate elasticsearch container that is being used now.

( Just for FYI, it didn't work at all at first. The solution was setting EX_ConnectionStrings__Elasticsearch to the actual IP of the server, not localhost:port )

But now i am kind of stuck, as of now i only get 401 or 403 responses, with fiddler or with the swaggerUI; on both: the IP:port and the tunneling domain name ...

maybe there is something wrong with my setup; as i almost never use docker really, here's the container info; Exceptionless Container Elasticsearch Container

[Crushedice]

Something is definitely weird

I tested out more things now, tried to switch the transport protocol of the tunnel from quic to http2 , didn't help.

but I also created 3 more api keys, one- from the dashboard, created normally, it was suspended after working for an hour or so one created with the API, and another one with the dashboard, which I disabled immediately. and ALL 3 are suspended, even though I did not use the api-created one, and even Disabled the other one beforehand.... it seems it has nothing even todo with any usage limits... which... really throws me off...

[
  {
    "id": "----------------------------------",
    "organization_id": "635c256d8c84b3001861afbb",
    "project_id": "635c256d8c84b3001861afbc",
    "scopes": [
      "client"
    ],
    "notes": "FallbackToken",
    "is_disabled": false,
    "is_suspended": true,
    "created_utc": "2022-10-28T18:54:38.0639116Z",
    "updated_utc": "2022-10-29T18:58:35.603809Z"
  },
  {
    "id": "-----------------------------------",
    "organization_id": "635c256d8c84b3001861afbb",
    "project_id": "635c256d8c84b3001861afbc",
    "scopes": [
      "user",
      "client"
    ],
    "notes": "ApiCreatedToken",
    "is_disabled": false,
    "is_suspended": true,
    "created_utc": "2022-10-29T18:57:34.3894015Z",
    "updated_utc": "2022-10-29T18:57:34.3904637Z"
  },
  {
    "id": "------------------------------------",
    "organization_id": "635c256d8c84b3001861afbb",
    "project_id": "635c256d8c84b3001861afbc",
    "scopes": [
      "client"
    ],
    "is_disabled": true,
    "is_suspended": true,
    "created_utc": "2022-10-29T18:58:57.3794739Z",
    "updated_utc": "2022-10-29T18:59:02.5639165Z"
  },
  {
    "id": "-------------------------------------------",
    "organization_id": "635c256d8c84b3001861afbb",
    "project_id": "635c256d8c84b3001861afbc",
    "scopes": [
      "client"
    ],
    "notes": "ApiTest",
    "is_disabled": false,
    "is_suspended": true,
    "created_utc": "2022-10-29T18:58:48.8288253Z",
    "updated_utc": "2022-10-29T18:59:24.8721593Z"
  }
]

[ejsmith]

Yeah, there has got to be some bug with code that we added for suspending tokens recently. I am going to dig into it today. Can you check to see if the org also got suspended or only the token?

[Crushedice]

Nope. the organization has not been suspended. Here the Elasticsearch info for the org

{
    "_index": "prod-organizations-v2",
    "_id": "635c256d8c84b3001861afbb",
    "_version": 32,
    "_seq_no": 31,
    "_primary_term": 5,
    "found": true,
    "_source": {
        "id": "635c256d8c84b3001861afbb",
        "name": "Rusticaland",
        "plan_id": "EX_UNLIMITED",
        "plan_name": "Unlimited",
        "plan_description": "Unlimited",
        "billing_change_date": "2022-10-28T18:54:37.6901438",
        "billing_changed_by_user_id": "635c25618c84b3001861afba",
        "billing_status": 0,
        "billing_price": 0,
        "max_events_per_month": -1,
        "bonus_events_per_month": 0,
        "retention_days": 180,
        "is_suspended": false,
        "has_premium_features": true,
        "max_users": -1,
        "max_projects": -1,
        "usage_hours": [
            {
                "date": "2022-10-28T19:00:00",
                "total": 17,
                "blocked": 0,
                "discarded": 0,
                "too_big": 0
            },
            {
                "date": "2022-10-28T20:00:00",
                "total": 25,
                "blocked": 0,
                "discarded": 0,
                "too_big": 0
            },
            {
                "date": "2022-10-29T12:00:00",
                "total": 11,
                "blocked": 0,
                "discarded": 0,
                "too_big": 0
            },
            {
                "date": "2022-10-29T19:00:00",
                "total": 89,
                "blocked": 0,
                "discarded": 0,
                "too_big": 0
            },
            {
                "date": "2022-10-29T20:00:00",
                "total": 34,
                "blocked": 0,
                "discarded": 0,
                "too_big": 0
            },
            {
                "date": "2022-10-29T22:00:00",
                "total": 14,
                "blocked": 0,
                "discarded": 0,
                "too_big": 0
            }
        ],
        "usage": [
            {
                "date": "2022-10-01T00:00:00",
                "limit": -1,
                "total": 190,
                "blocked": 0,
                "discarded": 0,
                "too_big": 0
            }
        ],
        "last_event_date_utc": "2022-10-29T22:20:13.4767519Z",
        "created_utc": "2022-10-28T18:54:37.6926766",
        "updated_utc": "2022-10-29T22:20:13.4771537Z",
        "is_deleted": false
    }
}

and the project

{
    "_index": "prod-projects-v2",
    "_id": "635c256d8c84b3001861afbc",
    "_version": 36,
    "_seq_no": 35,
    "_primary_term": 5,
    "found": true,
    "_source": {
        "id": "635c256d8c84b3001861afbc",
        "organization_id": "635c256d8c84b3001861afbb",
        "name": "Launcher",
        "is_configured": true,
        "configuration": {
            "version": 2,
            "settings": {
                "@@UserAgentBotPatterns": "*bot*,*crawler*,*spider*,*aolbuild*,*teoma*,*yahoo*",
                "@@log:*": "trace"
            }
        },
        "notification_settings": {
            "635c25618c84b3001861afba": {
                "send_daily_summary": true,
                "report_new_errors": true,
                "report_critical_errors": true,
                "report_event_regressions": true,
                "report_new_events": false,
                "report_critical_events": false
            }
        },
        "usage_hours": [
            {
                "date": "2022-10-28T19:00:00",
                "total": 17,
                "blocked": 0,
                "discarded": 0,
                "too_big": 0
            },
            {
                "date": "2022-10-28T20:00:00",
                "total": 25,
                "blocked": 0,
                "discarded": 0,
                "too_big": 0
            },
            {
                "date": "2022-10-29T12:00:00",
                "total": 11,
                "blocked": 0,
                "discarded": 0,
                "too_big": 0
            },
            {
                "date": "2022-10-29T19:00:00",
                "total": 89,
                "blocked": 0,
                "discarded": 0,
                "too_big": 0
            },
            {
                "date": "2022-10-29T20:00:00",
                "total": 34,
                "blocked": 0,
                "discarded": 0,
                "too_big": 0
            },
            {
                "date": "2022-10-29T22:00:00",
                "total": 14,
                "blocked": 0,
                "discarded": 0,
                "too_big": 0
            }
        ],
        "usage": [
            {
                "date": "2022-10-01T00:00:00",
                "limit": -1,
                "total": 190,
                "blocked": 0,
                "discarded": 0,
                "too_big": 0
            }
        ],
        "last_event_date_utc": "2022-10-29T22:20:13.4919481Z",
        "delete_bot_data_enabled": false,
        "next_summary_end_of_day_ticks": 638026884000000000,
        "created_utc": "2022-10-28T18:54:37.9628123",
        "updated_utc": "2022-10-29T22:20:13.4923334Z",
        "is_deleted": false
    }
}

in my user data is the field:

"roles": [
            "client",
            "user",
            "global"
        ],

Isn't "Admin" missing there, or am i wrong?

[ejsmith]

Thank you for the info! Global is the admin role.

[Crushedice]

Oh i see, then just FYI , if "Global" is the admin role, the token creation api tells you something different;

This is a helper action that makes it easier to create a token for a specific project. You may also specify a scope when creating a token. There are three valid scopes: client, user and admin.

Another thing, from the suspension of the tokens, even the unused ones, one can assume that its tied to some timer for cleanup, or some other actions, but I only found 2 instances where the exceptionless code modifies the tokens to "Is_suspended" . But as my organization is not suspended, it shouldn't be happening.

I looked into the elastic database, and got a question ; is it normal that "organizations" and "Projects" are labeled with "V2" where, the other ones are V1 ? -> Screenshot of Database

Also, a bit of a silly question: can't I "just" modify the entries in the database, and lock them to not get modified? Specifically the "is_suspended" field

[Crushedice]

Ok, I just now deployed the container with the app modus "Development" to get a usefull log.

And i think this one narrows it down pretty much :

{"log":"[20:30:00 INF] Found 0 suspended orgs","stream":"stdout","time":"2022-10-30T20:30:00.113443623Z"} ..................... ....................... {"log":"[20:30:00 INF] Marking 7 tokens as suspended","stream":"stdout","time":"2022-10-30T20:30:00.236620284Z"}

should be this method:

 private async Task MarkTokensSuspended(JobContext context) {
        var suspendedOrgs = await _organizationRepository.FindAsync(q => q.FieldEquals(o => o.IsSuspended, true).OnlyIds(), o => o.SearchAfterPaging().PageLimit(1000));
        _logger.LogInformation("Found {SuspendedOrgCount} suspended orgs", suspendedOrgs.Total);

        do {
            var updatedCount = await _tokenRepository.PatchAllAsync(q => q.Organization(suspendedOrgs.Hits.Select(o => o.Id)).FieldEquals(t => t.IsSuspended, false), new PartialPatch(new { is_suspended = true }));
            if (updatedCount > 0)
                _logger.LogInformation("Marking {SuspendedTokenCount} tokens as suspended", updatedCount);
        } while (!context.CancellationToken.IsCancellationRequested && await suspendedOrgs.NextPageAsync());
    }

[ejsmith]

Hmm. That's a huge help! I wonder if somehow the list of org ids being empty is causing all tokens to be returned??

[Crushedice]

I would have tried now, to create another org, suspend it, and see if a new api key gets suspended then.

But i don't think I can even make those changes with any API Call, same with the Web UI.

For the Method- I would "simply" put in a check for suspendedOrgs.Count()<=0 and if, don't even look for tokens ?

Because, as i see it, you are using the suspendedOrgs variable, despite being "empty" . Using the hits of the variable within the next LINQ method onsuspendedOrgs.Hits.Select

I get the feeling there could some mistakes happen.

[ejsmith]

Yeah, we can definitely add the count check. If you want, you can submit a PR since you found it. If not, I'll get it updated and also go check the repository implementation to make sure it's handling that correctly.

[niemyjski]

Great find!

[ejsmith]

This should be fixed in the lasted build. Thank you @Crushedice!