Closed jessegoodier closed 1 year ago
@jessegoodier In most cases, I believe these logs are harmless. Agree that if these logs are harmless, that logs should be minimized.
WRN ETL: Allocations: Federated Load intercept: federated load error: error reading combined federated file 'federated/combined/etl/bingen/allocations/1h/1679493600-1679497200': Read from S3 failed: The specified key does not exist.
I think this WRN occurs when the Primary tries to read an hourly ETL (1h
) from the combined
directory, but the file does not exist yet. If the epoch timestamp of that file is very close to present time, then it's because the Federator hasn't combined all clusters' hourly ETLs yet.
WRN Federator: Asset[1d]: Cannot read file: Error reading downstream ETL file at 'federated/ekw/etl/bingen/assets/1d/1679443200-1679529600': Read from S3 failed: The specified key does not exist.
I think this WRN occurs when the Federator is trying to read from the cloud storage bucket, but the Federated cluster hasn't yet pushed its ETLs to the bucket yet. Again, probably a harmless issue if the timestamp is close to present time.
@kaelanspatel Thoughts on above? Was this roughly correct?
That's basically right @thomasvn. The reason this show up so often is that there's no differentiation that can be made between the states that can lead to this WRN, whether benign states or not.
To clarify:
If this is causing undue confusion it could be moved to debug or deduped, but I think the intent of "this could possibly indicate an error state or might be fine, just letting you know" is correct for a WRN-level message.
Is there a way to hide these logs when expected? or reword to say, this can be expected when xxx?
yet another user today saw this and believes it is the cause of the issue.
@jessegoodier I can look into improving logging here. As mentioned, simplest solution would be to move these to DBG level logs, or deduping the logs.
+1 on moving this to debug if it's causing confusion. It could also be moved to an Inf level log, with a "less serious" sounding message e.g. Federated Load intercept: federated load: read combined federated file 'federated/combined/etl/bingen/assets/1h/1679493600-1679497200': Not reading from S3: federated data not present.
or something.
Describe the bug
All federated clusters that I have seen log this warning, which causes investigation into something that isn't a problem (or so I believe).
and
To Reproduce
Every federated ETL setup I have used has this WRN.
Expected behavior
If it truly isn't an issue, just hide it (maybe dbg only?)
logs:
|grep -i federator
logs:
|grep -i intercept