search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.86k stars 3.55k forks source link

Deletion via API responds 204 but does not delete #22591

Open sbhenderson opened 3 years ago

sbhenderson commented 3 years ago

Setup

Running Docker image influxdb:latest linux/amd64 version via Docker Desktop and Docker-compose on Windows 10 with WSL2 backend

Reported version info:

ts=2021-09-28T16:50:59.231871Z lvl=info msg="Welcome to InfluxDB" log_id=0WsKun7l000 version=2.0.8 commit=e91d41810f build_date=2021-08-13T18:22:30Z

Relevant docker-compose:

influxdb: image: influxdb:latest hostname: influxdb ports:

  • "8086:8086" volumes:
  • /run/desktop/mnt/host/c/infrastructure/influxdb:/var/lib/influxdb2 environment:
  • INFLUXD_LOG_LEVEL=debug

I have attached the files being used by InfluxDB. influxdb_files.zip

Username is user, password is password. Token that should be available is:

z2yImQ0GwQnfM5X2-SmP0l2ioWTx7w6xDtkhcacgEPVfUnLBwS79iYrm-I4IL-hvnWxOly8q-jY66CMVGiOhtw==

Influx CLI config creation:

.\influx.exe config create --config-name default --host-url http://localhost:8086 --org default --token z2yImQ0GwQnfM5X2-SmP0l2ioWTx7w6xDtkhcacgEPVfUnLBwS79iYrm-I4IL-hvnWxOly8q-jY66CMVGiOhtw== --active

Issue Replication

Using Influx CLI (version Influx CLI 2.1.1 (git: 535183b) build_date: 2021-09-22T18:32:23Z):

.\influx.exe delete --bucket Tenant1 --start '2021-08-01T00:00:00Z' --stop '2021-08-02T00:00:00Z'

Response via CLI was empty. Debug logs from InfluxDB show:

ts=2021-09-28T16:28:33.204170Z lvl=debug msg="user find by ID" log_id=0WsHS6Xl000 store=new took=0.016ms ts=2021-09-28T16:28:33.204340Z lvl=debug msg="org find" log_id=0WsHS6Xl000 store=new took=0.034ms ts=2021-09-28T16:28:33.204440Z lvl=debug msg="bucket find" log_id=0WsHS6Xl000 store=new took=0.070ms ts=2021-09-28T16:28:33.204473Z lvl=debug msg=Deleted log_id=0WsHS6Xl000 orgID=92f832d796c103c2 bucketID=b0cf74844a0697dd ts=2021-09-28T16:28:33.204607Z lvl=debug msg=Request log_id=0WsHS6Xl000 service=http method=POST host=localhost:8086 path=/api/v2/delete query="bucket=Tenant1&org=default" proto=HTTP/1.1 status_code=204 response_size=0 content_length=-1 referrer= remote=172.18.0.1:39768 user_agent=Date took=0.709ms body="{\"start\":\"2021-08-01T00:00:00Z\",\"stop\":\"2021-08-02T00:00:00Z\"}\n"

Result

I would expect the data to be indiscriminately deleted in the specified date range. However, running a query

.\influx.exe query 'from(bucket: \"Tenant1\") |> range(start: 2021-08-01T00:00:00Z, stop: 2021-08-02T00:00:00Z) |> filter(fn: (r) => r[\"_measurement\"] == \"14\")'

returns data within the specified range as seen below.

Result: _result Table: keys: [_start, _stop, _field, _measurement, history_type, qualityTag, type] _start:time _stop:time _field:string _measurement:string history_type:string qualityTag:string type:string _value:float _time:time

2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T05:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T06:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T07:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T08:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T09:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T10:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T11:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T12:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T13:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T14:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T15:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T16:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T17:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T18:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T19:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T20:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T21:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T22:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z VA 14 default 1 1 100 2021-08-01T23:00:00.000000000Z Table: keys: [_start, _stop, _field, _measurement, history_type, qualityTag, type] _start:time _stop:time _field:string _measurement:string history_type:string qualityTag:string type:string _value:int _time:time

2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T05:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T06:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T07:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T08:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T09:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T10:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T11:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T12:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T13:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T14:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T15:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T16:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T17:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T18:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T19:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T20:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T21:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T22:00:00.000000000Z 2021-08-01T00:00:00.000000000Z 2021-08-02T00:00:00.000000000Z quality 14 default 1 1 1 2021-08-01T23:00:00.000000000Z

Comments

I wanted to point out that this is extremely bizarre. I insert data into InfluxDB and have no problem clearing it out in most cases. This is the first time I'm running into behavior where it is simply not possible to delete the data even without a predicate. I was hoping for some sort of error message or log indicating the problem, but there's simply nothing from what I can see.

Please let me know how else I can help or provide more information. Thank you for your time.

sbhenderson commented 3 years ago

Finally, I would like to at least mention a coincidence with this issue. Recently, I had changed some logic to perform deletions of measurements in parallel allowing greater throughput considering that the API can either delete one measurement or all measurements. In addition, I'm not sure it should matter, but there are other interesting behavior as a result of this bug.

For example, by going to the dashboard and exploring, you can see that it does not believe there are any measurements in the given window image

Seems reasonable, but if you select _field first, you can then see the measurements. image

Could be entirely unrelated, but I thought it best to be forthcoming.

williamhbaker commented 3 years ago

Thanks for the excellent writeup @sbhenderson! I was able to reproduce exactly this behavior locally running on macos using a build of 2.0.9 using your data, so the behavior doesn't seem to be specific to windows/docker/etc. It also repros on a recent dev build so it doesn't look like it was fixed between 2.0.9 & now. This will take some digging into I think.

sbhenderson commented 3 years ago

@wbaker85 Please let me know if there's any additional information or details you need from me. I suspect that something I did caused the underlying store to get into this strange state, so I cannot advise if investigation should be on this "post-facto" store or if I should prepare a repro of getting it into this state. If you cannot find an obvious answer by the data itself, I may be able to produce a full replication from zero state which would hopefully elucidate the matter.

AnSchil commented 2 years ago

I got a similar issue using influx 2.1 via docker on my raspberry.

I delete points with the python sdk like this: start = datetime.fromisoformat(start_timestamp) - timedelta(weeks=2) stop = datetime.fromisoformat(stop_timestamp) + timedelta(seconds=9) try: client.delete_api().delete(start, stop, "", bucket=bucket_iot, org=org) client.close()

I log into that db every 10secs and send data from it every 60secs, after which I delete everything in it. I use influx to buffer messages when the network connection drops. After longer disconnects random parts of the db won't get deleted after a successful send. The wonky part is: newer points still get deleted every 60secs. It's nearly like the older points are delete protected. I have to manually delete the whole bucket to get rid of them. The number of points affected is looking random as well. No error message from inxludb. Every 60 sec that script will try to delete the affected points again with no success.

I'm really interested in the outcome of this. @sbhenderson did you make any progress?

yozik04 commented 1 year ago

Does not delete in 2.6.1

amotl commented 11 months ago

Hi there,

we are using the OCI image influxdb:2.7, and have been tripped by the same flaw that deleting data using the HTTP API apparently does not work.

We are using HTTPie to submit the delete request like this:

http POST "http://localhost:8086/api/v2/delete?org=example&bucket=testdrive" "Authorization:Token token" start="1877-09-21T00:12:43.145224194Z" stop="2162-04-11T23:47:16.854775806Z"

With kind regards, Andreas.

amotl commented 11 months ago

We need to revise our previous statement: Deleting data like outlined above, or alternatively using influx delete, works well, but data is still included when running influxd inspect export-lp. This is probably not related to this issue, so we will create a separate one.