search

CSIRO-enviro-informatics / loci.cat

Latest updates from the Loc-I Project
https://loci.cat
Other
0 stars 1 forks source link

Familiarise with DGGS Engine and python packaging #59

Closed jyucsiro closed 4 years ago

jevy-wangfei commented 4 years ago

Successfully make dggs_engine a python package. Ideas:

bellie888 commented 4 years ago

Hi Jevy, Personally I am happy with the structure. There is the "Engine" which is the dggs module, and the functions in it. The dggs Engine has a few dependencies in that same section, like 'distortion' , 'projectionwrapper', 'utils', and a couple of the pj files. The other pj_ files are not used at the moment as they are different ways to use geographic projections, but we only use one of them.

The "enablement_tagging" section is the scripts which use the 'Engine" and read in shapefiles to convert to DGGS format. I don't think these are part of the package but could be I suppose. We have many many scripts to do different DGGS jobs for different projects we has done over the last 2 years. DGGS is very flexible and can be used in many ways.

We are looking for an online open API that people can bring their shapefile or csv to and have it Auspix DGGS enabled. This would be point, line or polygon source data. The output results would allow us to interact with any other DGGS enabled data (in lookup tables or other AusPIX enabled data by relating on DGGS ID.

Happy to come over to CSIRO and share what I have been doing.

jevy-wangfei commented 4 years ago

Hi Joe, It's nice hearing from you. I'll keep the DGGS engine as it is, and use it as a package at the DGGS engine API service. As talked yesterday, we will speprate the DGGS engine API as an individal service hosted outside the loci-integration-api server, and the loci-integration-api will proxy requests to the DGGS engine API.

jyucsiro commented 4 years ago

We are looking for an online open API that people can bring their shapefile or csv to and have it Auspix DGGS enabled. This would be point, line or polygon source data. The output results would allow us to interact with any other DGGS enabled data (in lookup tables or other AusPIX enabled data by relating on DGGS ID.

Yes, it was clear from the conversation yesterday that an online OpenAPI service would be what we would aim to develop as an independent component. This would allow other apis, e.g. loci-integration-api, and clients to call it to run those 'enablement' use cases. @jevy-wangfei and the team will look into this.

The "enablement_tagging" section is the scripts which use the 'Engine" and read in shapefiles to convert to DGGS format. I don't think these are part of the package but could be I suppose. We have many many scripts to do different DGGS jobs for different projects we has done over the last 2 years. DGGS is very flexible and can be used in many ways.

It'd be good to develop those scripts into callable functions as part of the dggs engine package. Perhaps as a utils or enablement submodule? What do you think @bellie888?

bellie888 commented 4 years ago

@jevy-wangfei I will see if GA can provide the DGGS API hosting space, maybe on AWS if you like. That way it is a collaborative effort and one we are already invested in. What do you think? I will ask around to see if it is possible.

bellie888 commented 4 years ago

@jevy-wangfei Hi again, Shane Crossman is pretty keen to support the DGGS API at GA. We will provide space for you if that works for you? Cheers, Joe

bellie888 commented 4 years ago

@jyucsiro Hi Jonathan, yes those enablement scripts are as simple as I could make them - to help any developer get started on how to import a shapefile and have it DGGS enabled. The scripts output as csv or shapefile too. Shapefile is good for visualisation. - for those that want it.

Yes they will need to be converted into callable scripts, importing the shape or csv file, and also the resolution required - with someway of downloading the result. Many thanks Joe

bellie888 commented 4 years ago

@jyucsiro It'd be good to develop those scripts into callable functions as part of the dggs engine package. Perhaps as a utils or enablement submodule? What do you think @bellie888?

Yes you could replace the scripts in the current enablement folder with your more developed versions. Thanks

bellie888 commented 4 years ago

@jyucsiro @jevy-wangfei So can I check that you are happy for GA to provide hosting on AWS? Hope it doesn't take too long to set up at GA. I can start working on it right away, but I don't want is to hold up progress. Let me know.

jyucsiro commented 4 years ago

@jyucsiro @jevy-wangfei So can I check that you are happy for GA to provide hosting on AWS? Hope it doesn't take too long to set up at GA. I can start working on it right away, but I don't want is to hold up progress. Let me know.

@bellie888 - yes that would be great! the current thinking is developing a flask app that wraps the AusPIX DGGS engine and provides web apis and deployment via a docker container. That way, we can develop it and you can deploy it on AWS. Would that suit the GA AWS hosting arrangement?

bellie888 commented 4 years ago

Talking to Irina this morning a Docker container method was discussed with Will and Shane and it was "approved" as a suitable method.