search

beanumber / etl

R package to facilitate ETL operations
127 stars 21 forks source link

etl_load for spatial data #55

Open cZachJohnson opened 4 years ago

cZachJohnson commented 4 years ago

@beanumber have you applied this framework to spatial data? I noticed that the macleish uses spacial data but from my understanding it doesn't load the spatial data to the SQL database. I have made a few attempts to but I seem to run into an issue when it checks for csv.

beanumber commented 4 years ago

Correct, it does not work with spatial data. I think PostgreSQL supports spatial data, but I am not sure about the other implementations.

A way forward could certainly be to detect that the data is spatial in the transform phase, and then leverage any spatial support in dbplyr in the load phase.

cZachJohnson commented 4 years ago

Okay so that would require writing both an etl_transform.package and the etl_load.package? I noticed there was a mention of templates for both of these, I was unable to find them when I ran the create_etl_package function? I appreciate the help and thank you for creating this package.

beanumber commented 4 years ago

Correct.

There is only a template for etl_extract() created by the function, because that is the only one that is always necessary. etl_transform.default() will move all CSVs from raw to load, and etl_load.default() will load all CSVs in load into eponymous tables in the database. If you need other behavior (which it sounds like you will), then you will have to write your own methods.

cZachJohnson commented 4 years ago

Okay thank you for the help.