search

kubecost / features-bugs

A public repository for filing of Kubecost feature requests and bugs. Please read the issue guidelines before filing an issue here.
0 stars 0 forks source link

[Bug] Duckdb corruption after 2.2.5 upgrade #103

Open DerekTBrown opened 4 months ago

DerekTBrown commented 4 months ago

Kubecost Version

2.2.5

Kubernetes Version

1.25

Kubernetes Platform

Other (specify in description)

Description

After a 2.2.5 upgrade, I see the aggregator container failing to start with the following message:

2024-06-03T18:58:50.534412476Z ERR error doing initial open of DB: error opening db at path /var/configs/waterfowl/duckdb/v0_9_2/kubecost.duckdb.write: setting up migrations: opening '/var/configs/waterfowl/duckdb/v0_9_2/kubecost.duckdb.write': could not open database: IO Error: Corrupt database file: computed checksum 2550608162518328766 does not match stored checksum 6531264450723538710 in block
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x177fe55]

Steps to reproduce

  1. Launch kubecost and wait.

Expected behavior

  1. No pod crashes.

Impact

No response

Screenshots

No response

Logs

No response

Slack discussion

No response

Troubleshooting

DerekTBrown commented 4 months ago

This looks similar to the following issues, but those are fixed:

AjayTripathy commented 4 months ago

Hi @DerekTBrown we're working on a smoother way to handle this, but for now you should be able to delete and recreate the persistent volume and restart the pod to get out of this state.

DerekTBrown commented 3 months ago

Hi @DerekTBrown we're working on a smoother way to handle this, but for now you should be able to delete and recreate the persistent volume and restart the pod to get out of this state.

Done, and that did seem to resolve the issue.

Is the plan to just have something that deletes the cache if it becomes corrupted?

AjayTripathy commented 3 months ago

@cliffcolvin we're pretty sure this is getting addressed in 2.3+ right?

DeepakRai94 commented 3 months ago

I had the same issue, but it started working after I recreated the aggregator-db persistent volume.

passionInfinite commented 3 months ago

Confirmed same issue with v2.2.4.

teevans commented 3 months ago

Hey there, this should be resolved in our 2.3 release. We're planning on releasing 2.3.2 sometime today or tomorrow and recommend upgrading to that when it's ready!

timchenko-a commented 2 weeks ago

Not sure if this is the same exact issue, but we've hit something similar with 2.3.5:

panic: failed to create ingestor: Ingestor: error creating db: setting up migrations: opening '/var/configs/waterfowl/duckdb/v0_10_3/kubecost.duckdb.write': database/sql/driver: could not open database: duckdb error: IO Error: Corrupt database file: computed checksum 4178360413824115490 does not match stored checksum 16005271743778032503 in block at location 34877440
AjayTripathy commented 2 weeks ago

Could you try and recreate the aggregator db volume and let me know if that works @timchenko-a