NYCPlanning / db-data-library

📚 Data Library
https://nycplanning.github.io/db-data-library/library/index.html
MIT License
0 stars 1 forks source link

Allow subpath for local zip archives, use gdal exceptions sooner in process #411

Closed fvankrieken closed 1 year ago

fvankrieken commented 1 year ago

First part should be self-explanatory.

For second part, I've been doing some testing locally with cpdb shape files. Currently, if a zipped shapefile is invalid in some way, this is discovered by loading the dataset without using exceptions, then raising an error with assert if the dataset is None. It seems a bit more helpful to actually let gdal throw the error while loading the dataset instead (though in this case it does just say only slightly more verbosely that the zip file is not in a supported format). A little nervous about side-effects of this - do we have existing known "errors" that should be ignored? Should we always be throwing errors? I would think yes - a little odd that right now we only allow exceptions at the translate portion, not the loading of the dataset.

@damonmcc @alexrichey @AmandaDoyle

@athursland and @DeaBardhoshi just fyi

damonmcc commented 1 year ago

yea my reflex is that we'd rather always throw errors. I recently added gdal.UseExceptions() where it is once I learned this very significant fact:

By default, the GDAL and OGR Python bindings do not raise exceptions when errors occur. The GDAL team acknowledges that Python programmers expect exceptions to be enabled by default, but says that exceptions are disabled by default to preserve backward compatibility.

maybe I didn't put it where you're moving it to because I was so focused on gdal.VectorTranslate issues and was scared lol