search

MAAP-Project / Community

Issue for MAAP (Zenhub)
2 stars 1 forks source link

Create a base DPS image for users #966

Closed sujen1412 closed 2 weeks ago

sujen1412 commented 6 months ago

Currently, users are using (as base image) a pre-built base_image from https://github.com/MAAP-Project/maap-workspaces/tree/main/base_images. When an algorithm is submitted from registration we are including additional steps for creating a user ops and re-doing all permissions which takes up a lot of time during registration.

TODO: Move up the work required at minimum by DPS to base images to save time during registration.

Future (not covered in this ticket) : How to optimize space taken by these images.

rtapella commented 5 months ago

how does this relate to Stacks?

sujen1412 commented 3 months ago

First pass, we will have a basic conda image as base image with no other packages installed.

sujen1412 commented 3 months ago

@chuckwondo @wildintellect Please try mas.dit.maap-project.org/root/maap-workspaces/base_images/dps:dit. Built from https://github.com/MAAP-Project/maap-workspaces/blob/dps_base_image/dps_base_image/Dockerfile

chuckwondo commented 2 months ago

@sujen1412 and I wrangled things in DIT and saw the GEDI Subsetter image go from 10.4GB using the python:v4.0.0 base image down to 4.31GB using the pared down dps image, so we saw ~60% reduction in size.

chuckwondo commented 2 months ago

@sujen1412, is there more you need to do before you can push the stripped down DPS base image to OPS?

sujen1412 commented 2 months ago

Yes, there are a few more tests I need to run to ensure that jobs run successfully on the workers. This is done to test user permissions on directories that are mounted within a container when the job runs.