search

NOAA-OWP / hydrovis

Other
11 stars 8 forks source link

Create SD GP Service: Automate setup of custom python environment on EGIS Servers #446

Open CoreyKrewson-NOAA opened 1 year ago

CoreyKrewson-NOAA commented 1 year ago

This is to help issue #218. In order for the geoprocessing tool to publish with the script that Tyler has created, a custom python environment has to be added to the EGIS servers.

We need to find a way to automate this setup and be customizable from a requirement.txt file or similar so that we can change the environment as needed.

ESRI doc for using a custom conda environment for ArcGIS Server: https://enterprise.arcgis.com/en/server/latest/publish-services/windows/deploying-custom-python-packages.htm NOTE: The only explicit additional package is boto3 (i.e. boto3 and its dependencies)

CoreyKrewson-NOAA commented 1 year ago

@TylerSchrag-NOAA I setup this issue as a support for your ticket about the GP tool

shawncrawley commented 2 months ago

I believe this must be done via creating a new Amazon Machine Image (AMI) for the EC2 machine that the conda environment is set up on. This new AMI would of course include everything the base AMI includes, plus the conda environment.

The TI EC2 of interest: https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#InstanceDetails:instanceId=i-020524988d12b580f

EdisonOrellana-NOAA commented 3 weeks ago

Getting ArcGIS server to work with a custom python environment has proven extremely difficult if not impossible. Esri documentation claims that this is possible, but not in practice.

Details

The exact issue is unidentified, but it lies in that using a custom python environment on ArcGIS server seems to cause the [code](https://github.com/NOAA-OWP/hydrovis/blob/ce4cc1e8be923ea69574b52769f82ac9a99842fb/Source/Visualizations/aws_loosa/utils/arcgis_gps_mapx_to_sd.py) to go unexecuted. Using the default environment fixes the issue.

@shawncrawley suggested avoiding the issue by using the default ArcGIS python environment so as not to introduce more python package requirements (boto3+ others)

This has proven to be successful by making subprocess.run() calls to aws cli which is installed by default on EC2 machines.

This issue can remain under consideration since it is something that should work in theory and would be useful in the future, but at this point this alternative solution in service of #218 will work.