Open Ahmed-Soliman96 opened 1 month ago
@Ahmed-Soliman96 can you please paste the contents of your flow file here?
Also, does this occur when you don't use the Runner API? i.e. just with the CLI?
@madhur-ob Thank you for your response. Unfortunately, I cannot paste the flow file content, cause it's a confidential script. But I can guarantee that this error doesn't occur when running the script from CLI, it occurs only with Runner API.
Hi @Ahmed-Soliman96 I wasn't able to reproduce it on my side. From a first glance, it seems like a Postgres issue aka wrong column type. Maybe the schema of the postgres table you use is the issue behind the scenes?
But regardless, I am not asking for the exact contents of the flow file, but something similar perhaps..
which makes use of these parameters: max_num_splits=2000, max_workers=64
, amongst others..
Essentially, something reproducible after removing all the confidential parts... which emulates what you are trying to do..
When I run a flow:
with metadata="local" I get this error:
https://github.com/Netflix/metaflow/blob/a617aa816dde0651d481fdfb6bbafe067667dc4a/metaflow/plugins/metadata/service.py#L471
This error never happens when I run without the metadata = "local" parameter.