search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.65k stars 3.54k forks source link

Very difficult error; unexpected error writing points to database: [shard 12] unexpected end of JSON input, Docker Compose, need help #24959

Open JounaydG opened 4 months ago

JounaydG commented 4 months ago

I have been searching and experimenting for hours now. The problem is that in my docker compose project i cannot send data from nodered to influxdb anymore. Everythig used to work fine. But, after letting the containers run for 3 days, it stopped working. Docker daemon couldnt start up anymore. After research, I came to the conclusion that there was not enough storage left, which ended up being the case. So, i checked what was using so much space and truncated the docker container that used up most space (it was 46Gb somehow and i have 55GB in total). This somehow solved the issue of docker daemon not starting up. However, after succesfully starting up the containers I got the following error message in nodered.

error message in nodered, regarding influxdbout node; "HttpError: unexpected error writing points to database: [shard 12] unexpected end of JSON input"

I checked the container logs of influxdb; repeated lines in the influxdb container logs; ts=2024-05-05T19:38:23.123011Z lvl=info msg="Write failed" log_id=0ozXsPcG000 service=storage-engine service=write shard=12 error="[shard 12] unexpected end of JSON input" ts=2024-05-05T19:38:23.123100Z lvl=error msg="Recorder handler error" log_id=0ozXsPcG000 error="[shard 12] unexpected end of JSON input" ts=2024-05-05T19:38:31.533197Z lvl=info msg="Write failed" log_id=0ozXsPcG000 service=storage-engine service=write shard=12 error="[shard 12] unexpected end of JSON input" ts=2024-05-05T19:38:33.095098Z lvl=info msg="Write failed" log_id=0ozXsPcG000 service=storage-engine service=write shard=12 error="[shard 12] unexpected end of JSON input" ts=2024-05-05T19:38:33.123268Z lvl=info msg="Write failed" log_id=0ozXsPcG000 service=storage-engine service=write shard=12 error="[shard 12] unexpected end of JSON input" ts=2024-05-05T19:38:33.123348Z lvl=error msg="Recorder handler error" log_id=0ozXsPcG000 error="[shard 12] unexpected end of JSON input" ts=2024-05-05T19:38:33.189497Z lvl=info msg="Write failed" log_id=0ozXsPcG000 service=storage-engine service=write shard=12 error="[shard 12] unexpected end of JSON input" ts=2024-05-05T19:38:40.309153Z lvl=info msg="Write failed" log_id=0ozXsPcG000 service=storage-engine service=write shard=12 error="[shard 12] unexpected end of JSON input"

removing all containers by doing docker compose down and then up again sadly doesn't work... I feel like im so fcked right now, because this is part of my bachelor's project... I didn't expect this to come up

extra info; im working on a debian linux 11 environment. I only have acces to this computer via ssh

davidby-influx commented 2 months ago

It appears that when you truncated the docker container you corrupted (or truncated) the files in the database.