Can't connect Loki to Grafana in k8s

Describe the bug Hi, I want connect Loki to Grafana, it's running in Kubernetes. I have running Loki in k8s by Helm Chart with this values

deploymentMode: SingleBinary
loki:
  commonConfig:
    replication_factor: 1
  storage:
    type: 'filesystem'
  schemaConfig:
    configs:
    - from: "2024-01-01"
      store: tsdb
      index:
        prefix: loki_index_
        period: 24h
      object_store: filesystem # we're storing on filesystem so there's no real persistence here.
      schema: v13
singleBinary:
  replicas: 1
read:
  replicas: 0
backend:
  replicas: 0
write:
  replicas: 0

And have running Grafana by Helm Chart with defaults values.

When I want add Loki to datasources by k8s DNS http://loki-gateway.loki.svc.cluster.local/ I receive this error Unable to connect with Loki. Please check the server logs for more details..

I check Grafana logs and see this

logger=context userId=1 orgId=1 uname=admin t=2024-06-07T21:52:24.907808918Z level=info msg="Request Completed" method=GET path=/api/datasources/uid/cdo3bm9uqbhmoa/health status=400 remote_addr=127.0.0.1 time_ms=10004 duration=10.004715235s size=106 referer=http://127.0.0.1:3000/connections/datasources/edit/cdo3bm9uqbhmoa handler=/api/datasources/uid/:uid/health status_source=server
logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=checkHealth pluginId=loki dsName=loki dsUID=cdo3bm9uqbhmoa uname=admin fromAlert=false t=2024-06-07T21:52:28.272753526Z level=info msg="Prepared request to Loki" duration=74.659µs queriesLength=1 stage=prepareRequest runInParallel=false
logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=checkHealth pluginId=loki dsName=loki dsUID=cdo3bm9uqbhmoa uname=admin fromAlert=false t=2024-06-07T21:52:28.304783632Z level=error msg="Error received from Loki" duration=31.305741ms stage=databaseRequest statusCode=410 contentLength= start=1970-01-01T00:00:01Z end=1970-01-01T00:00:04Z step=1s query=vector(1)+vector(1) queryType=instant direction=backward maxLines=0 supportingQueryType=none lokiHost=loki-gateway.loki.svc.cluster.local:80 lokiPath=/loki/api/v1/query status=error error="<html>\n  <head>\n    <meta http-equiv='refresh' content='0; url=http://loki-gateway.loki.svc.cluster.local/' />\n  </head>\n</html>\n" statusSource=downstream
logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth t=2024-06-07T21:52:28.305113568Z level=error msg="Loki health check failed" error="error from loki: <html>\n  <head>\n    <meta http-equiv='refresh' content='0; url=http://loki-gateway.loki.svc.cluster.local/' />\n  </head>\n</html>\n"
logger=context userId=1 orgId=1 uname=admin t=2024-06-07T21:52:28.305249615Z level=info msg="Request Completed" method=GET path=/api/datasources/uid/cdo3bm9uqbhmoa/health status=400 remote_addr=127.0.0.1 time_ms=40 duration=40.979212ms size=106 referer=http://127.0.0.1:3000/connections/datasources/edit/cdo3bm9uqbhmoa handler=/api/datasources/uid/:uid/health status_source=server
logger=cleanup t=2024-06-07T21:55:33.356920878Z level=info msg="Completed cleanup jobs" duration=47.654942ms

Whats wrong?

grafana / loki

Can't connect Loki to Grafana in k8s #13179