DROP SERIES is not reflected in SHOW TAG VALUES

[arindamchoudhury]

Hi,

I am using influxdb 1.6.2

When I do DROP SERIES FROM "measurement" WHERE "tag_name" = 'tag-value'

this gets reflected in SHOW SERIES FROM "measurement"

But

SHOW TAG VALUES FROM "measurement" WITH KEY = "tag_name"

still shows the tag-value

[e-dard]

Hi @arindamchoudhury I was able to reproduce this on the tsi1 index, but no the inmem index. Thanks for the report.

> create database db
> insert cpu,region=west value=1
> insert cpu,region=east value=1
>
> DROP SERIES FROM "cpu" WHERE "region"='west'
>
> SHOW SERIES FROM "cpu"
key
---
cpu,region=east
>
> SHOW TAG VALUES FROM "cpu" WITH KEY = "region"
name: cpu
key    value
---    -----
region east
region west
>
>

[lpic10]

Any news on this? I use SHOW TAG VALUES FROM system WITH KEY=host on grafana for host template, but that finds a lot of hosts already deleted from influxdb

[benbjohnson]

@lpic10 Sorry for the delay on this. We're looking into it.

@e-dard The issue stems from this line here:

https://github.com/influxdata/influxdb/blob/master/tsdb/engine/tsm1/engine.go#L1475

The inmem index has all data in-memory so it can easily compute the new measurement index data. However, Rebuild() on tsi1 is a no-op because that would be an expensive operation.

One option would be to check for the existence of all key/value pairs on all the series deleted. That also seems a bit expensive but I'm not sure how else we handle it.

[ushuz]

Same issue for me on 1.6.4 with inmem index. There are several other reports in #5092.

SHOW TAG VALUES WITH KEY = "some-tag" returns tags from dropped series, however SHOW TAG VALUES WITH KEY = "some_tag" WHERE another_tag = 'another_tag' won't return those dropped tags.

---

A workaround for me: SHOW TAG VALUES WITH KEY = "some_tag" WHERE some_tag != ''

[lpic10]

@lpic10 Sorry for the delay on this. We're looking into it.

@e-dard The issue stems from this line here:

https://github.com/influxdata/influxdb/blob/master/tsdb/engine/tsm1/engine.go#L1475

The inmem index has all data in-memory so it can easily compute the new measurement index data. However, Rebuild() on tsi1 is a no-op because that would be an expensive operation.

One option would be to check for the existence of all key/value pairs on all the series deleted. That also seems a bit expensive but I'm not sure how else we handle it.

Is there some workaround, eg. manually rebuilding the tsi indexes?

[cocampbe]

I just noticed this. Prior to running 1.7.3 these values would be removed after dropping all series.

> drop series where host = 'dhcp-10-96-109-139' >

> show series where host = 'dhcp-10-96-109-139' >

When I do a query to show tag values, I still see the host.

> show tag values from "system" with key = "host" name: system key value --- ----- host dhcp-10-96-109-112 host dhcp-10-96-109-125 host dhcp-10-96-109-134 host dhcp-10-96-109-139

[cocampbe]

Well. Restarting the db ended up being a work around for me.

[juliencorsini]

Hi there, Any planned fix on this issue? I am also seeing this behavior in 1.7.4.

[stuartmacfarlane]

Same issue here on influxdb-1.7.3-1.x86_64 running pretty much default settings in the config file. Restarting the service clears down the lingering tags.

[arindamchoudhury]

Is there any manual way to fix it? without restarting.

[lishunan246]

The same with version 1.7.4 on Ubuntu x64.

[jpldev]

Any news about this? Thanks

[Pshellvon]

Same thing with InfluxDB 1.7.4 on Debian 9

[dandv]

Confirmed in 1.7.7 on Ubuntu 18.04 with XFS. Tried to reproduce with a sample measurement, but it seems to happen only for larger databases.

Related: https://community.influxdata.com/t/commad-to-delete-a-particular-series-from-influxdb/5600/18?u=dandv

UPDATE: systemctl restart influxd fixed the problem - the zombie tags are gone.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[arindamchoudhury]

this still happens. please dont make it stale.

[sbm17]

Is there any resolution for this? Tried to drop series and restart the db. I still see the values.

[Esity]

@russorat is there a reason why the stale-exempt label was removed?

[nicolas17]

This issue is obviously present and easily reproducible. It's not requiring more information from users. Why do you need "activity" on the issue? It's not stale until it's fixed.

Don't make me post a comment every month to keep it open.

[russorat]

@Esity @nicolas17 we got rid of stalebot so there's no need for that label anymore.

[afenioux]

have you tried to rebuild tsi index? (it worked for me)

https://docs.influxdata.com/influxdb/v1.7/administration/rebuild-tsi-index/

[doglex]

A workaround: clone to a temp database and then clone back.

create database temp use udp select into temp..:MEASUREMENT from tail group by drop database udp create database udp use temp select into udp..:MEASUREMENT from tail group by

[arindamchoudhury]

Hi,

just a question out of curiosity, it was included in milestone 1.8.0 which is already released. will it be reconsidered for another milestone?

[hanej]

Is there an update on this issue?

[dominikkv]

Yo. I'm asking for a friend.

[dbk-rabel]

Any news on this issue? Restarting InfluxDB as a workaround does not seem to work for us. Some old tag values are still present. others seem to have gone. Rebuilding the tsi index regularly is no option on a productive database under heavy load.

[rdxmb]

same problem here with influxdb 1.8.1

[barnabasbusa]

Is this issue resolved with 2.x? Rebuilding is not an option for large production server, yet I just want to drop some tags.

[wz2b]

Is this issue resolved with 2.x? Rebuilding is not an option for large production server, yet I just want to drop some tags.

I don't know but I don't think so ... but I'm wondering of anybody can answer this question: can you solve this by rebuilding the index with https://docs.influxdata.com/influxdb/v1.8/tools/influx_inspect/#buildtsi ?

[q1234q1234]

Is this issue resolved with 2.x? Rebuilding is not an option for large production server, yet I just want to drop some tags.

I don't know but I don't think so ... but I'm wondering of anybody can answer this question: can you solve this by rebuilding the index with https://docs.influxdata.com/influxdb/v1.8/tools/influx_inspect/#buildtsi ?

Nope with version 1.8.10-1. Even after 5th or 6th TSI rebuild + multiple restarts only 1 from 4 tags dissapeared.

[Amerousful]

It also doesn't work in version 2.1.1 I tried delete and build tsi - the result is the same :(

[Glokeru]

I can confirm this. Tags do not dissapear.

[antonio-castellon]

Anyone takes this issue seriously? because seems like dont. It's important to be able to do at least one of this two actions ones someone delete data-series: (1) delete the tags from the global index
(2) a sentence that returns only tags with existing data associated Best regards A.

[antonio-castellon]

as I saw, a restart of the database fix the global index, but... any way to relaunch the index of the TAGs without to restart the server?

[fireba11]

wtf guys, you can't just "not update the index because it's too expensive", especially with tsi index where not even a restart fixes it. the only (hopefully working) workaround is shut down influxdb, delete indexes manually and have them rebuild? we are using tsi indexes because in-memory ram costs escalated too much and slowed restarts to a crawl. so rebuilding isn't really a good workaround, also it might take so long our telegrafs start loosing data waiting for influxdb to become available again.

at least implement a proper index rebuild command

[arindamchoudhury]

@fireba11 I can understand your frustration

[Glokeru]

How should i do a tsi-rebuild in influx 2.3, to get rid of old tag values? It is not documented. Something like this?

1.) Shutdown influxdb 2.) Remove all index folders 3.) Run influxd inspect build-tsi 4.) Start influxdb

Thanks