search

AllenNeuralDynamics / aind-data-asset-indexer

MIT License
0 stars 0 forks source link

Scheduled S3 crawler task is failing in data asset indexer ECS prod environment #29

Closed yosefmaru closed 5 months ago

yosefmaru commented 5 months ago

ValueError: You are trying to merge on object and float64 columns for key 's3_prefix'. If you wish to proceed you should use pd.concat

More can be found on cloudwatch logs in prod:

https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/$252Fecs$252FAindDataAssetIndexerTD/log-events/ecs$252FDataAssetIndexer$252Fd67c97b6dec94bc09d13546595570eb3$3Fstart$3D1705377646996

yosefmaru commented 5 months ago

Fixed by changing the env variables in the ECS task definition. We should probably add a check in the script for catching empty dataframes. https://github.com/AllenNeuralDynamics/aind-data-asset-indexer/issues/30