Open aeisenbarth opened 1 year ago
zanieb
commented
1 year ago @aeisenbarth thanks for the report!
Do you have any clue what integer we're passing that would be this large?
This looks like a wont-fix upstream https://github.com/ijl/orjson/issues/116
aeisenbarth
commented
1 year ago I couldn't get a debugger stop at orjson_dumps in prefect server. But I printed the to-be-serialized object and it seems to be a FlowRunResponse from a previous flow run:
v = {
'id': UUID('73ce02d0-6b1e-40b7-b37e-b08991b2ea6d'),
'created': DateTime(2023, 3, 9, 15, 21, 28, 762739, tzinfo = Timezone('UTC')),
'updated': DateTime(2023, 3, 9, 15, 21, 39, 828928, tzinfo = Timezone('UTC')),
'name': 'batch_test.py::test_run_flow_from_deployment[flows0-parameters0-True-True]',
'flow_id': UUID('d221605b-a6f0-4b92-bbcd-b3eddf03399a'),
'state_id': UUID('8633aaaf-37a1-410c-9959-14b3c7d7adb6'),
'deployment_id': None,
'work_queue_name': 'default',
'flow_version': '996b9d88712d1f8baddf7bb958c44972',
'parameters': {
'max_limit': 179769313486231570000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
},
'idempotency_key': None,
'context': {},
'empirical_policy': {
'max_retries': 0,
'retry_delay_seconds': 0.0,
'retries': 0,
'retry_delay': 0,
'pause_keys': set(),
'resuming': False
},
'tags': [],
'parent_task_run_id': None,
'state_type': StateType.COMPLETED,
'state_name': 'Completed',
'run_count': 1,
'expected_start_time': DateTime(2023, 3, 9, 15, 21, 28, 747364, tzinfo = Timezone('UTC')),
'next_scheduled_start_time': None,
'start_time': DateTime(2023, 3, 9, 15, 21, 39, 797307, tzinfo = Timezone('UTC')),
'end_time': DateTime(2023, 3, 9, 15, 21, 39, 827905, tzinfo = Timezone('UTC')),
'total_run_time': datetime.timedelta(microseconds = 30598),
'estimated_run_time': datetime.timedelta(microseconds = 30598),
'estimated_start_time_delta': Duration(seconds = 11, microseconds = 49943),
'auto_scheduled': False,
'infrastructure_document_id': UUID('0f04b26b-4ae5-4be5-b993-7a0aa6e424c9'),
'infrastructure_pid': 'computer:594873',
'created_by': None,
'work_pool_name': 'default-agent-pool',
'state': {
'id': UUID('8633aaaf-37a1-410c-9959-14b3c7d7adb6'),
'type': StateType.COMPLETED,
'name': 'Completed',
'timestamp': DateTime(2023, 3, 9, 15, 21, 39, 827905, tzinfo = Timezone('UTC')),
'message': None,
'data': {
'type': 'literal',
'value': None
},
'state_details': {
'flow_run_id': UUID('73ce02d0-6b1e-40b7-b37e-b08991b2ea6d'),
'task_run_id': None,
'child_flow_run_id': None,
'scheduled_time': None,
'cache_key': None,
'cache_expiration': None,
'untrackable_result': False,
'pause_timeout': None,
'pause_reschedule': False,
'pause_key': None,
'refresh_cache': None
}
}
}It seems the culprit is in our data. We had some code where a float64 (!) parameter defaults to the maximum value. This seemed to be valid Python code and did not appear as a problem as long as we used the sqlite database. But in combination with PostgreSQL, the float is somehow converted/stored/parsed as an integer, which is obviously bigger than 64-bit.
Here is a minimal example flow:
import sys
from prefect import flow
@flow
def my_flow(max_limit: float):
return None
my_flow(max_limit=sys.float_info.max)We can find a solution to avoid such large floats in our code. But still there is somewhere an underlying problem. I would at least expect the float to be still a float (possibly the maximum supported by the database, or even overlowed).
First check
Bug summary
When using Prefect with a PostgreSQL database and loading the UI flow-runs page, I get a TypeError "Integer exceeds 64-bit range" caused by https://github.com/ijl/orjson/issues/301.
I just wanted to mention it so that it is tracked here. The orjson bug report mentions a work-around which would require patching serialization code in Prefect.
My current workaround is not to configure a PostgreSQL database but resort to the default sqlite until the upstream issue is solved.
Reproduction
Error
Versions
Additional context
No response