too many outstanding requests

[stefangweichinger]

At first: thanks for sharing your work!

I followed your howto ( https://voidquark.com/parsing-nextcloud-audit-logs-with-grafana-loki/ ) and successfully get nextcloud logs into Loki.

The dashboard works in general but often gives me: "Too many outstanding requests"

Could you share a working loki-config maybe, I wonder if I miss something there?

My environment:

• nc-26.0.5 in docker
• grafana/loki:2.8.4
• grafana/promtail:2.8.4
• grafana/grafana-oss:10.1.1

all hosted on Debian-12.1.

This is a rather small nextcloud-instance with ~50 users.

Additional thoughts:

• I managed to use the loki docker driver to directly pipe nextcloud-logs (type "syslog") into loki without writing and reading the logfiles (basically avoiding promtail). This looks like an improvement to me and might be worth a look. But the queries in the dashboard would have to be adjusted as well. I wonder if this might help to improve the overall perfomance.

[voidquark]

Hello @stefangweichinger

Unfortunately, this issue falls outside the scope of this repository. It is related to the Loki configuration. I highly recommend referring to the documentation for guidance on resolving this issue. You can find discussions related to this problem here: https://github.com/grafana/loki/issues/5123. This issue may provide insights into why you're encountering this problem and how you can optimize your Loki configuration.

Regarding piping Nextcloud logs: Currently, I do not have plans to create multiple variations of dashboards. However, you are welcome to download the dashboard and customize it to suit your specific setup and requirements.

[wforumw]

I have added this to my loki config.yml

query_range:
  parallelise_shardable_queries: false

limits_config:
  split_queries_by_interval: 0

Restart your loki service

[stefangweichinger]

I have added this to my loki config.yml

query_range:
  parallelise_shardable_queries: false

limits_config:
  split_queries_by_interval: 0

Restart your loki service

Thanks, trying this now. I assume that's not the full loki-config, right?

[stefangweichinger]

Hello @stefangweichinger

Unfortunately, this issue falls outside the scope of this repository. It is related to the Loki configuration. I highly recommend referring to the documentation for guidance on resolving this issue. You can find discussions related to this problem here: grafana/loki#5123. This issue may provide insights into why you're encountering this problem and how you can optimize your Loki configuration.

Thanks, browsing through right now and trying ...

Regarding piping Nextcloud logs: Currently, I do not have plans to create multiple variations of dashboards. However, you are welcome to download the dashboard and customize it to suit your specific setup and requirements.

Sure, thanks

[stefangweichinger]

Just to get things right:

Might query errors also play a role here:

prometheus-loki-1  | level=error ts=2023-09-06T12:58:04.645439838Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 query="{env=~\"production\", job=~\"nextcloud\", instance=~\"cloud\\\\.my\\\\.tld\"} | json | level=\"4\" | line_format \"➡️☠️ IP {{.remoteAddr}} {{.method}} {{.url }} with ? {{.user}} ? {{.message }} \"" err="context canceled"

[voidquark]

Just to get things right:

Might query errors also play a role here:

prometheus-loki-1  | level=error ts=2023-09-06T12:58:04.645439838Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 query="{env=~\"production\", job=~\"nextcloud\", instance=~\"cloud\\\\.my\\\\.tld\"} | json | level=\"4\" | line_format \"➡️☠️ IP {{.remoteAddr}} {{.method}} {{.url }} with ? {{.user}} ? {{.message }} \"" err="context canceled"

Could you please share the raw log line from Loki explorer, specifically from the Nextcloud audit log with level 4? It's important to note that this issue may not be directly related to your Loki configuration but could be connected to the dashboard itself.

Additionally, could you check if there are any log lines displayed in the Nextcloud Recent Log panel? Please make sure to redact any sensitive information before sharing.

Also, it would be helpful to verify if the count of FATAL Log Lines in the panel matches the number of Fatal messages in the Nextcloud Recent Log. For example, if the FATAL Log Lines panel shows 4, you should also find 4 instances of Fatal messages in the Nextcloud Recent Log.

You can also share these FATAL messages in a screenshot, which would provide further clarity and aid in resolving the issue.

[stefangweichinger]

@voidquark thanks for the feedback, I will try to check things tomorrow morning

[stefangweichinger]

@voidquark just a quick and partial look:

No "FATAL Log Lines" in the panel. Are they even logged with 'loglevel' => 1, in config.php ?

Yes, there are log lines displayed in the Nextcloud Recent Log panel.

[voidquark]

No "FATAL Log Lines" in the panel. Are they even logged with 'loglevel' => 1, in config.php ?

Yes, they are indeed logged with 'loglevel' => 1 in the config.php file. You can find more information on Nextcloud logging in the Nextcloud documentation.

Yes, there are log lines displayed in the Nextcloud Recent Log panel.

Great to hear that there are log lines being displayed in the Nextcloud Recent Log panel. However, it appears that you are not seeing any FATAL messages with the ☠️ emoji.

Please check Loki explorer for any FATAL log lines that contain level=4

[stefangweichinger]

@voidquark I assume that this means that these FATAL log lines would have to be in either nextcloud.log or audit.log, right?

I grep the 2 files and only find such lines in nextcloud.log. That one is rather long and only level 3:

{"reqId":"SBmBXNEX2rwYCWyjZcGt","level":3,"time":"2023-09-07T05:28:59+00:00","remoteAddr":"10.0.0.20","user":"--","app":"core","method":"PROPFIND","url":"/remote.php/dav/files/wexxx/","message":"Failed to connect to the database: An exception occurred in the driver: SQLSTATE[08006] [7] connection to server at \"postgres_db\" (172.27.0.3), port 5432 failed: FATAL:  sorry, too many clients already","userAgent":"Mozilla/5.0 (Windows) mirall/3.9.3stable-Win64 (build 20230818) (Nextcloud, windows-10.0.22621 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"26.0.5.1","exception":{"Exception":"Doctrine\\DBAL\\Exception","Message":"Failed to connect to the database: An exception occurred in the driver: SQLSTATE[08006] [7] connection to server at \"postgres_db\" (172.27.0.3), port 5432 failed: FATAL:  sorry, too many clients already","Code":7,"Trace":[{"file":"/var/www/html/3rdparty/doctrine/dbal/src/Connection.php","line":1531,"function":"connect","class":"OC\\DB\\Connection","type":"->","args":[]},{"file":"/var/www/html/3rdparty/doctrine/dbal/src/Connection.php","line":1029,"function":"getWrappedConnection","class":"Doctrine\\DBAL\\Connection","type":"->","args":[]},{"file":"/var/www/html/lib/private/DB/Connection.php","line":262,"function":"executeQuery","class":"Doctrine\\DBAL\\Connection","type":"->","args":["SELECT * FROM \"oc_appconfig\"",[],[],null]},{"file":"/var/www/html/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php","line":345,"function":"executeQuery","class":"OC\\DB\\Connection","type":"->","args":["SELECT * FROM \"oc_appconfig\"",[],[]]},{"file":"/var/www/html/lib/private/DB/QueryBuilder/QueryBuilder.php","line":280,"function":"execute","class":"Doctrine\\DBAL\\Query\\QueryBuilder","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppConfig.php","line":418,"function":"execute","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppConfig.php","line":184,"function":"loadConfigValues","class":"OC\\AppConfig","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppConfig.php","line":374,"function":"getApps","class":"OC\\AppConfig","type":"->","args":[]},{"file":"/var/www/html/lib/private/legacy/OC_App.php","line":976,"function":"getValues","class":"OC\\AppConfig","type":"->","args":[false,"installed_version"]},{"file":"/var/www/html/lib/private/Server.php","line":731,"function":"getAppVersions","class":"OC_App","type":"::","args":[]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":171,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":138,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":171,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Memcache\\Factory",true]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":65,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":193,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":171,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":138,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\ICacheFactory"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":171,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\ICacheFactory",true]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":97,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\ICacheFactory",true]},{"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":84,"function":"array_map","args":[["Closure"],[["ReflectionParameter","config"],["ReflectionParameter","oldDispatcher"],"*** sensitive parameters replaced ***",["ReflectionParameter","eventDispatcher"]]]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":124,"function":"buildClass","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":[["ReflectionClass","OC\\User\\Manager"]]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":142,"function":"resolve","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Manager"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":171,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Manager",true]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":65,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\User\\Manager"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":193,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Manager"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":171,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":138,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\IUserManager"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":171,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IUserManager",true]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":65,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\IUserManager"]},{"file":"/var/www/html/lib/private/Server.php","line":557,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IUserManager"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":171,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":138,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\User\\Session"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":171,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Session",true]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":65,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\User\\Session"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":193,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Session"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":171,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":138,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\IUserSession"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":171,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IUserSession",true]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":65,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\IUserSession"]},{"file":"/var/www/html/lib/private/Server.php","line":1406,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IUserSession"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":171,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":138,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\ISession"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":171,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\ISession",true]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":65,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\ISession"]},{"file":"/var/www/html/lib/private/Server.php","line":1281,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\ISession"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":171,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":138,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\Security\\CSRF\\TokenStorage\\SessionStorage"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":171,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Security\\CSRF\\TokenStorage\\SessionStorage",true]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":97,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\Security\\CSRF\\TokenStorage\\SessionStorage",true]},{"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":84,"function":"array_map","args":[["Closure"],[["ReflectionParameter","tokenGenerator"],"*** sensitive parameters replaced ***"]]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":124,"function":"buildClass","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":[["ReflectionClass","OC\\Security\\CSRF\\CsrfTokenManager"]]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":142,"function":"resolve","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Security\\CSRF\\CsrfTokenManager"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":171,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Security\\CSRF\\CsrfTokenManager",true]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":65,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\Security\\CSRF\\CsrfTokenManager"]},{"file":"/var/www/html/lib/private/Server.php","line":1062,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Security\\CSRF\\CsrfTokenManager"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":171,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":138,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\IRequest"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":171,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IRequest",true]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":65,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\IRequest"]},{"file":"/var/www/html/lib/private/Server.php","line":1547,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IRequest"]},{"file":"/var/www/html/lib/private/legacy/OC_Template.php","line":320,"function":"getRequest","class":"OC\\Server","type":"->","args":[]},{"file":"/var/www/html/remote.php","line":92,"function":"printExceptionErrorPage","class":"OC_Template","type":"::","args":[["Doctrine\\DBAL\\Exception"],500]},{"file":"/var/www/html/remote.php","line":174,"function":"handleException","args":[["Doctrine\\DBAL\\Exception"]]}],"File":"/var/www/html/lib/private/DB/Connection.php","Line":140,"CustomMessage":"--"}}

I don't find anything with grep -i fatal /mnt/nextcloud-data/nextcloud.log | grep "level\":4". And no FATAL log lines in audit.log.

[voidquark]

No, this is not considered a level 4. It's actually categorized as an Error level because it's marked as level 3, even though it displays a fatal message. The crucial factor here is level 4.

Could you please open Grafana, navigate to Explore, and execute the following query:

{job=~"YOURJOB", instance=~"INSTANCE"} | json | level="4" | line_format "➡️☠️ IP {{.remoteAddr}} {{.method}} {{.url }} with 👤 {{.user}} 💬 {{.message }}"

Please ensure that you've replaced the values in the job and instance fields. If you encounter any warnings or errors during parsing, please provide the specific line and error details. It would be most helpful if you could include a screenshot of the log message directly with all labels.

[stefangweichinger]

Did the mentioned query, no results. For a check I edited the query to level 3, this gives results.

edit: my obfuscating missed a bit ;-) no problem

[voidquark]

Alright, in this case, we can consider it not to be an issue because there are no level 4 messages, which, from Nextcloud's perspective, are categorized as FATAL. I will go ahead and close the issue since it is not a problem.

[stefangweichinger]

Hm, might be.

Still seeing this in the docker logs:

prometheus-loki-1  | level=error ts=2023-09-07T13:23:35.096152064Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 query="{instance=~\"cloud\\\\.my\\\\.tld\", job=~\"nextcloud\", instance=~\"cloud\\\\.ikw-amstetten\\\\.at\"} | json | level=\"4\" | line_format \"➡️☠️ IP {{.remoteAddr}} {{.method}} {{.url }} with ? {{.user}} ? {{.message }} \"" err="context canceled"

[stefangweichinger]

Seems I found a working loki-config, things look smooth now. Thanks.

[NightDragon1]

Seems I found a working loki-config, things look smooth now. Thanks.

Could you please share it?

[stefangweichinger]

Seems I found a working loki-config, things look smooth now. Thanks.

Could you please share it?

Sure, but give me some time. Busy days ... and I have to look it up and rethink the context.

[stefangweichinger]

Seems I found a working loki-config, things look smooth now. Thanks.

Could you please share it?

here my full loki-config. Maybe it helps.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

limits_config:
  split_queries_by_interval: 24h
  max_query_parallelism: 100

ruler:
  alertmanager_url: http://alertmanager:9093

query_scheduler:
  max_outstanding_requests_per_tenant: 4096

querier:
  max_concurrent: 100

frontend:
    max_outstanding_per_tenant: 4096

It definitely could be improved, I assume ... hints welcome.

[NightDragon1]

here my full loki-config. Maybe it helps.

Thank you mate!