kubecost / cost-analyzer-helm-chart

Kubecost helm chart
http://kubecost.com/install
Apache License 2.0
488 stars 418 forks source link

log issue: federated etl WRN ETL: Allocations: Federated Load intercept: The specified key does not exist. #2073

Closed jessegoodier closed 1 year ago

jessegoodier commented 1 year ago

Describe the bug

All federated clusters that I have seen log this warning, which causes investigation into something that isn't a problem (or so I believe).

WRN ETL: Allocations: Federated Load intercept: federated load error: error reading combined federated file 'federated/combined/etl/bingen/allocations/1h/1679493600-1679497200': Read from S3 failed: The specified key does not exist.

and

WRN Federator: Asset[1d]: Cannot read file: Error reading downstream ETL file at 'federated/ekw/etl/bingen/assets/1d/1679443200-1679529600': Read from S3 failed: The specified key does not exist.

To Reproduce

Every federated ETL setup I have used has this WRN.

Expected behavior

If it truly isn't an issue, just hide it (maybe dbg only?)

logs: |grep -i federator

2023-03-22T14:31:35.949238977Z INF Federator: Asset: Beginning merge of 9 changed file(s) into combined federated set
2023-03-22T14:31:35.975545019Z WRN Federator: Asset[1d]: Cannot read file: Error reading downstream ETL file at 'federated/ekw/etl/bingen/assets/1d/1679443200-1679529600': Read from S3 failed: The specified key does not exist.
2023-03-22T14:31:36.116260249Z INF Federator: Asset[1d]: Successfully merged files for '1679443200-1679529600' from federated clusters
2023-03-22T14:31:36.287497377Z WRN Federator: Asset[1h]: Cannot read file: Error reading downstream ETL file at 'federated/ekw/etl/bingen/assets/1h/1679468400-1679472000': Read from S3 failed: The specified key does not exist.
2023-03-22T14:31:36.540487678Z INF Federator: Asset[1h]: Successfully merged files for '1679468400-1679472000' from federated clusters
2023-03-22T14:31:36.642937455Z WRN Federator: Asset[1h]: Cannot read file: Error reading downstream ETL file at 'federated/ekw/etl/bingen/assets/1h/1679472000-1679475600': Read from S3 failed: The specified key does not exist.
2023-03-22T14:31:36.787026767Z INF Federator: Asset[1h]: Successfully merged files for '1679472000-1679475600' from federated clusters
2023-03-22T14:31:36.889844193Z WRN Federator: Asset[1h]: Cannot read file: Error reading downstream ETL file at 'federated/ekw/etl/bingen/assets/1h/1679475600-1679479200': Read from S3 failed: The specified key does not exist.
2023-03-22T14:31:37.009529239Z INF Federator: Asset[1h]: Successfully merged files for '1679475600-1679479200' from federated clusters

logs: |grep -i intercept

2023-03-22T14:14:56.418064566Z WRN ETL: Assets: Federated Load intercept: federated load error: error reading combined federated file 'federated/combined/etl/bingen/assets/1h/1679493600-1679497200': Read from S3 failed: The specified key does not exist.
2023-03-22T14:23:40.724940577Z WRN ETL: Assets: Federated Load intercept: federated load error: error reading combined federated file 'federated/combined/etl/bingen/assets/1h/1679493600-1679497200': Read from S3 failed: The specified key does not exist.
2023-03-22T14:24:56.624044883Z WRN ETL: Assets: Federated Load intercept: federated load error: error reading combined federated file 'federated/combined/etl/bingen/assets/1h/1679493600-1679497200': Read from S3 failed: The specified key does not exist.
thomasvn commented 1 year ago

@jessegoodier In most cases, I believe these logs are harmless. Agree that if these logs are harmless, that logs should be minimized.

WRN ETL: Allocations: Federated Load intercept: federated load error: error reading combined federated file 'federated/combined/etl/bingen/allocations/1h/1679493600-1679497200': Read from S3 failed: The specified key does not exist.

I think this WRN occurs when the Primary tries to read an hourly ETL (1h) from the combined directory, but the file does not exist yet. If the epoch timestamp of that file is very close to present time, then it's because the Federator hasn't combined all clusters' hourly ETLs yet.

WRN Federator: Asset[1d]: Cannot read file: Error reading downstream ETL file at 'federated/ekw/etl/bingen/assets/1d/1679443200-1679529600': Read from S3 failed: The specified key does not exist.

I think this WRN occurs when the Federator is trying to read from the cloud storage bucket, but the Federated cluster hasn't yet pushed its ETLs to the bucket yet. Again, probably a harmless issue if the timestamp is close to present time.

@kaelanspatel Thoughts on above? Was this roughly correct?

kaelanspatel commented 1 year ago

That's basically right @thomasvn. The reason this show up so often is that there's no differentiation that can be made between the states that can lead to this WRN, whether benign states or not.

To clarify:

If this is causing undue confusion it could be moved to debug or deduped, but I think the intent of "this could possibly indicate an error state or might be fine, just letting you know" is correct for a WRN-level message.

jessegoodier commented 1 year ago

Is there a way to hide these logs when expected? or reword to say, this can be expected when xxx?

yet another user today saw this and believes it is the cause of the issue.

thomasvn commented 1 year ago

@jessegoodier I can look into improving logging here. As mentioned, simplest solution would be to move these to DBG level logs, or deduping the logs.

kaelanspatel commented 1 year ago

+1 on moving this to debug if it's causing confusion. It could also be moved to an Inf level log, with a "less serious" sounding message e.g. Federated Load intercept: federated load: read combined federated file 'federated/combined/etl/bingen/assets/1h/1679493600-1679497200': Not reading from S3: federated data not present. or something.