Open Zeitsperre opened 6 years ago
The post I was referring to yesterday @huard.
Again, I think it's a good idea to do this. Speak to @dbyrns about the priority to give to this. If it can be partially done short term and helps with testing, that might be moved upward in the priority list.
@Zvax mentions in #84 an issue with the rendering time for some elaborate/complex multi-part polygons being really slow, while @tlogan2000 has told me before that there are a few contiguity/topology errors for some of the shapefiles, causing some analyses to outright fail. This all relates to some discussion raised in https://github.com/Ouranosinc/pavics-sdi/issues/32 about data inconsistencies between deployments.
As a potential fix, can I suggest that that we create a list of open data set urls that can be downloaded and extracted during the PAVICS installation process? I feel that this would solve some challenges with perform tests on deployments and would also give users an "out-of-the-box" functionality when they build the PAVICS stack. This would also clear up the rendering time issues in #84 as we can look for more simplified shapefiles than we currently have on some deployments.
It can also place us in the clear when it comes to data distribution privileges, as we can append the data licenses for whichever data sets we include by default by downloading them at the same time or placing them within the github/docker image or within the docs (Any suggestions where would be a good place to download/place the licenses?). Any proprietary data sets that users add to PAVICS will be their issue to manage via permissions systems.
The data I have in mind will likely consist of the following:
Polygons:
Lines:
Points:
Climate Grids:
Once I have a list of data set URLs, what would be the process of adding a method of downloading and extracting the data to a folder that the crawler would look at by default using the docker/docker-compose method? Are there data sets that missing/should be removed? Thoughts?