Closed DistributedDoge closed 8 months ago
Thanks for opening the issue @DistributedDoge!
As I haven't used the metadata
fields, I didn't realize they were being treated as Python objects.
I'll update the ingestion code to make everything proper JSON!
Meanwhile, do you mind sending a PR upstream with your changes? No pressure as I was thinking on redoing the Python assets soon.
Indexer has several objects (
projects, rounds, applications
) withmetadata
served as well-formed, nested json.When inspecting database tables
raw_projects, raw_rounds, raw_round_applications
from the latest release (dbt.duckdb
file) somewhere along the pipelinemetadata (varchar)
column in each of those tables was transformed into pythonic format that can no longer satisfy JSON parser, but needs to be passed to pythonast.literal_eval()
:I wanted to be able to run a simple Duckdb query to extract interesting fields like one below. This won't work as-is, because single quotes break JSON parser:
To achieve that quickly I modified asset generation step as follows (
ggdp/assets.py
). Goal is to ensure that Pandas dataframe generated by dagster contains JSON string instead of a collection of Python objects. This seems to be working on my fork. For future reference I should probably just replacemetadata
column instead of duplicating it.I am writing this to suggest that upstream could also benefit from having
metadata
forraw_
tables in JSON format.