Open bkozura opened 1 year ago
clairelin135
commented
1 year ago @bkozura -- thanks for all the details here. Oddly, when I run an asset with mysql storage on the versions you mentioned, I don't see an error. This error is also surprising given that the multi partition key class is a string subclass, so it should be able to serialize...
Would you mind also sharing a code sample of the asset you're running, and validating that those versions above apply to your deployment?
pirackr
commented
1 year ago @clairelin135 I got the same issue and I can reproduce this with basically any multi-partition keys. Something like this and run it as an asset will trigger the issue
monthly = MonthlyPartitionsDefinition(
start_date="2023-01-01"
)
other_partitions = DynamicPartitionsDefinition(name="other") # or other can be a static partition
partition = MultiPartitionsDefinition(
{"date": monthly, "other": other_partitions }
)
@asset(partitions_def=partition, name="asset")
def my_asset(context: OpExecutionContext):
return []
job = build_schedule_from_partitioned_job(job=my_asset, default_status=DefaultScheduleStatus.RUNNING)I think SQLAlchemy doesn't understand the MultiPartitionKey as a string, even though it is subclass of str. This could potentially be a compatibility issue between mysql-connector and SQLAlchemy - I read SQLAlchemy docs and they don't recommend to use mysql-connector as the driver, it is not even included in their integration test pipeline.
A quick workaround can get me bypass this:
--- a/python_modules/dagster/dagster/_core/storage/event_log/sql_event_log.py
+++ b/python_modules/dagster/dagster/_core/storage/event_log/sql_event_log.py
if event.dagster_event.partition:
partition = event.dagster_event.partition
+ partition = str(partition) if partition is not None else None
# https://stackoverflow.com/a/54386260/324449
return SqlEventLogStorageTable.insert().values(
run_id=event.run_id,
timestamp=datetime.utcfromtimestamp(event.timestamp),
step_key=step_key,
asset_key=asset_key_str,
partition=partition,
)But I assume it's just a workaround.
clairelin135
commented
1 year ago @pirackr got it, thanks for the detailed explanation. Glad you found a workaround--I'll update the code when I get the chance to handle this, likely with the same workaround you mentioned :)
Dagster version
version 1.3.12
What's the issue?
Good morning When I try to run an asset materialization with multipartitions (one monthly, one static). I am running into this unexpected error:
This happens on both the dev and prod envs. It does not happen on my local environment.
What did you expect to happen?
Asset materialization runs
How to reproduce?
package versions
Deployment type
Other
Deployment details
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.