agrc / palletjack

A library for updating AGOL data from various external sources
https://agrc.github.io/palletjack/palletjack/
MIT License
12 stars 0 forks source link

Feat: Overwrite (truncate and load) existing feature service #13

Closed jacobdadams closed 2 years ago

jacobdadams commented 2 years ago

Implements a full truncate and load of a layer within a feature service using data from a spatially-enabled dataframe. Figured out how to use .append() for hopefully more performant, reliable edits.

codecov[bot] commented 2 years ago

Codecov Report

Merging #13 (693b74e) into main (0014c2e) will increase coverage by 2.87%. The diff coverage is 98.90%.

@@            Coverage Diff             @@
##             main      #13      +/-   ##
==========================================
+ Coverage   83.61%   86.49%   +2.87%     
==========================================
  Files           4        4              
  Lines         409      496      +87     
  Branches       56       65       +9     
==========================================
+ Hits          342      429      +87     
  Misses         65       65              
  Partials        2        2              
Impacted Files Coverage Δ
src/palletjack/updaters.py 91.20% <98.90%> (+3.01%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 0014c2e...693b74e. Read the comment docs.

jacobdadams commented 2 years ago

Ok @steveoh, I think I've built in some robustness. Let me know what you think.

jacobdadams commented 2 years ago

This is going to be in a cloud function right? Does that writing need to persist in a different way because of where it will be run?

I don't think so- you pass in the directory name to truncate_and_load_feature_service, so it will be the client skid's responsibility to make sure that directory is writeable. I'm assuming that GCS will allow you to pass a path like that? Or does it need to be more bucket-oriented?

Also, I tweaked _retry() to eliminate the need for all the lambda calls. I've tried testing it but it looks like the latest AGOL update introduced a bug that causes the featurelayer.manager.truncate(asynchronous=True, wait=True) call to fail. In touch with support, waiting for a fix. Once I've got that I'll test it again with AGOL to verify everything works as intended.

jacobdadams commented 2 years ago

Ok, AGOL bug fixed, looks like everything works as intended. @steveoh, I think the only open item is the question about file location. If that looks good to you, I think we're good to rebase and merge.

steveoh commented 2 years ago

I'm assuming that GCS will allow you to pass a path like that? Or does it need to be more bucket-oriented?

Let's try it and see. It very well may end up being something like gs://bucket or something. Or it will need to be reworked to push an object to GCS.