search

NYCPlanning / data-engineering

Primary repository for NYC DCP's Data Engineering team
14 stars 0 forks source link

Move Metadata to new Repo #888

Closed alexrichey closed 3 weeks ago

alexrichey commented 3 weeks ago

It lives here now!

alexrichey commented 3 weeks ago

Is there any code right now that expects these metadata files to live where they currently are?

@fvankrieken Nope! The Metadata files are going to be packaged up with the product+datasets on s3, so once we implement packaging, the location will start to matter.

damonmcc commented 2 weeks ago

@alexrichey

The Metadata files are going to be packaged up with the product+datasets on s3, so once we implement packaging, the location will start to matter.

so right now for a CI or local use of python3 -m dcpy.cli lifecycle distribute from_s3 ... to succeed, a metadata file has to be manually put in the relevant folder?

alexrichey commented 2 weeks ago

@damonmcc Yes, the metadata file will be part of the package on s3, e.g. image

That's the metadata file that the distribution CLI will use (though you override for testing purposes).