influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.9k stars 3.55k forks source link

DROP SERIES is not reflected in SHOW TAG VALUES #10285

Open arindamchoudhury opened 6 years ago

arindamchoudhury commented 6 years ago

Hi,

I am using influxdb 1.6.2

When I do DROP SERIES FROM "measurement" WHERE "tag_name" = 'tag-value'

this gets reflected in SHOW SERIES FROM "measurement"

But

SHOW TAG VALUES FROM "measurement" WITH KEY = "tag_name"

still shows the tag-value

e-dard commented 6 years ago

Hi @arindamchoudhury I was able to reproduce this on the tsi1 index, but no the inmem index. Thanks for the report.

> create database db
> insert cpu,region=west value=1
> insert cpu,region=east value=1
>
> DROP SERIES FROM "cpu" WHERE "region"='west'
>
> SHOW SERIES FROM "cpu"
key
---
cpu,region=east
>
> SHOW TAG VALUES FROM "cpu" WITH KEY = "region"
name: cpu
key    value
---    -----
region east
region west
>
>
lpic10 commented 5 years ago

Any news on this? I use SHOW TAG VALUES FROM system WITH KEY=host on grafana for host template, but that finds a lot of hosts already deleted from influxdb

benbjohnson commented 5 years ago

@lpic10 Sorry for the delay on this. We're looking into it.

@e-dard The issue stems from this line here:

https://github.com/influxdata/influxdb/blob/master/tsdb/engine/tsm1/engine.go#L1475

The inmem index has all data in-memory so it can easily compute the new measurement index data. However, Rebuild() on tsi1 is a no-op because that would be an expensive operation.

One option would be to check for the existence of all key/value pairs on all the series deleted. That also seems a bit expensive but I'm not sure how else we handle it.

ushuz commented 5 years ago

Same issue for me on 1.6.4 with inmem index. There are several other reports in #5092.

SHOW TAG VALUES WITH KEY = "some-tag" returns tags from dropped series, however SHOW TAG VALUES WITH KEY = "some_tag" WHERE another_tag = 'another_tag' won't return those dropped tags.


A workaround for me: SHOW TAG VALUES WITH KEY = "some_tag" WHERE some_tag != ''

lpic10 commented 5 years ago

@lpic10 Sorry for the delay on this. We're looking into it.

@e-dard The issue stems from this line here:

https://github.com/influxdata/influxdb/blob/master/tsdb/engine/tsm1/engine.go#L1475

The inmem index has all data in-memory so it can easily compute the new measurement index data. However, Rebuild() on tsi1 is a no-op because that would be an expensive operation.

One option would be to check for the existence of all key/value pairs on all the series deleted. That also seems a bit expensive but I'm not sure how else we handle it.

Is there some workaround, eg. manually rebuilding the tsi indexes?

cocampbe commented 5 years ago

I just noticed this. Prior to running 1.7.3 these values would be removed after dropping all series.

> drop series where host = 'dhcp-10-96-109-139' >

> show series where host = 'dhcp-10-96-109-139' >

When I do a query to show tag values, I still see the host.

> show tag values from "system" with key = "host" name: system key value --- ----- host dhcp-10-96-109-112 host dhcp-10-96-109-125 host dhcp-10-96-109-134 host dhcp-10-96-109-139

cocampbe commented 5 years ago

Well. Restarting the db ended up being a work around for me.

juliencorsini commented 5 years ago

Hi there, Any planned fix on this issue? I am also seeing this behavior in 1.7.4.

stuartmacfarlane commented 5 years ago

Same issue here on influxdb-1.7.3-1.x86_64 running pretty much default settings in the config file. Restarting the service clears down the lingering tags.

arindamchoudhury commented 5 years ago

Is there any manual way to fix it? without restarting.

lishunan246 commented 5 years ago

The same with version 1.7.4 on Ubuntu x64.

jpldev commented 5 years ago

Any news about this? Thanks

Pshellvon commented 5 years ago

Same thing with InfluxDB 1.7.4 on Debian 9

dandv commented 5 years ago

Confirmed in 1.7.7 on Ubuntu 18.04 with XFS. Tried to reproduce with a sample measurement, but it seems to happen only for larger databases.

Related: https://community.influxdata.com/t/commad-to-delete-a-particular-series-from-influxdb/5600/18?u=dandv

UPDATE: systemctl restart influxd fixed the problem - the zombie tags are gone.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

arindamchoudhury commented 5 years ago

this still happens. please dont make it stale.

sbm17 commented 4 years ago

Is there any resolution for this? Tried to drop series and restart the db. I still see the values.

Esity commented 4 years ago

@russorat is there a reason why the stale-exempt label was removed?

nicolas17 commented 4 years ago

This issue is obviously present and easily reproducible. It's not requiring more information from users. Why do you need "activity" on the issue? It's not stale until it's fixed.

Don't make me post a comment every month to keep it open.

russorat commented 4 years ago

@Esity @nicolas17 we got rid of stalebot so there's no need for that label anymore.

afenioux commented 4 years ago

have you tried to rebuild tsi index? (it worked for me)

https://docs.influxdata.com/influxdb/v1.7/administration/rebuild-tsi-index/

doglex commented 4 years ago

A workaround: clone to a temp database and then clone back.

create database temp use udp select into temp..:MEASUREMENT from tail group by drop database udp create database udp use temp select into udp..:MEASUREMENT from tail group by

arindamchoudhury commented 4 years ago

Hi,

just a question out of curiosity, it was included in milestone 1.8.0 which is already released. will it be reconsidered for another milestone?

hanej commented 4 years ago

Is there an update on this issue?

dominikkv commented 4 years ago

Yo. I'm asking for a friend.

dbk-rabel commented 3 years ago

Any news on this issue? Restarting InfluxDB as a workaround does not seem to work for us. Some old tag values are still present. others seem to have gone. Rebuilding the tsi index regularly is no option on a productive database under heavy load.

rdxmb commented 3 years ago

same problem here with influxdb 1.8.1

barnabasbusa commented 3 years ago

Is this issue resolved with 2.x? Rebuilding is not an option for large production server, yet I just want to drop some tags.

wz2b commented 2 years ago

Is this issue resolved with 2.x? Rebuilding is not an option for large production server, yet I just want to drop some tags.

I don't know but I don't think so ... but I'm wondering of anybody can answer this question: can you solve this by rebuilding the index with https://docs.influxdata.com/influxdb/v1.8/tools/influx_inspect/#buildtsi ?

q1234q1234 commented 2 years ago

Is this issue resolved with 2.x? Rebuilding is not an option for large production server, yet I just want to drop some tags.

I don't know but I don't think so ... but I'm wondering of anybody can answer this question: can you solve this by rebuilding the index with https://docs.influxdata.com/influxdb/v1.8/tools/influx_inspect/#buildtsi ?

Nope with version 1.8.10-1. Even after 5th or 6th TSI rebuild + multiple restarts only 1 from 4 tags dissapeared.

Amerousful commented 2 years ago

It also doesn't work in version 2.1.1 I tried delete and build tsi - the result is the same :(

Glokeru commented 2 years ago

I can confirm this. Tags do not dissapear.

antonio-castellon commented 2 years ago

Anyone takes this issue seriously? because seems like dont. It's important to be able to do at least one of this two actions ones someone delete data-series: (1) delete the tags from the global index
(2) a sentence that returns only tags with existing data associated Best regards A.

antonio-castellon commented 2 years ago

as I saw, a restart of the database fix the global index, but... any way to relaunch the index of the TAGs without to restart the server?

fireba11 commented 2 years ago

wtf guys, you can't just "not update the index because it's too expensive", especially with tsi index where not even a restart fixes it. the only (hopefully working) workaround is shut down influxdb, delete indexes manually and have them rebuild? we are using tsi indexes because in-memory ram costs escalated too much and slowed restarts to a crawl. so rebuilding isn't really a good workaround, also it might take so long our telegrafs start loosing data waiting for influxdb to become available again.

at least implement a proper index rebuild command

arindamchoudhury commented 2 years ago

@fireba11 I can understand your frustration

Glokeru commented 2 years ago

How should i do a tsi-rebuild in influx 2.3, to get rid of old tag values? It is not documented. Something like this?

1.) Shutdown influxdb 2.) Remove all index folders 3.) Run influxd inspect build-tsi 4.) Start influxdb

Thanks