search

MorphoCloud / MorphoCloudWorkflow

Reusable GitHub Workflows to manage JetStream2 backed on-demand virtual machines
BSD 2-Clause "Simplified" License
1 stars 1 forks source link

Install slicer to MyData & increase MyData size to 100GB #54

Closed muratmaga closed 3 days ago

muratmaga commented 2 weeks ago

There are now about a dozen cases (in different situations), I had to delete and recreate the instance for one reason or another. In these situations it is a bit frustrating to start from the scratch for Slicer installations (you might have customizations, you might have installed extension, python packages etc)...

solution if possible install the Slicer tree under /media/volume/MyData/Slicer (possibly also the MC files referred in this issue https://github.com/MorphoCloud/MorphoCloudWorkflow/issues/51 to MyData)

Given that this may reduce the storage size considerably (depending on the packages installed), I propose we move the storage to 100GB. There has been at least one situation where we couldn't fit the data to an empty 60GB storage.

muratmaga commented 6 days ago

We will have to use preview version of Slicer on the cloud. I will add this comment for tracking purposes.

jcfr commented 6 days ago

Since the volume needs to be attached to successfully install Slicer, there are a few possible approaches:

  1. Attach the volume directly from within the ansible role before installing Slicer
  2. Skip the installation of Slicer from within the ansible role and proceed only after the volume is attached.
  3. Before attaching, rename /media/volume/MyData/ to /media/volume/MyData-tmp, then attach volume, copy folders into it and delete /media/volume/MyData-tmp
muratmaga commented 5 days ago

I do not have any preference. 1st one seems simple enough (i.e., change the order of the events).

jcfr commented 5 days ago

After evaluating the proposed solutions, I found that approach (3) was the most straightforward to implement. I have implemented this approach and am currently finalizing testing.

In the current implementation, before attaching the volume, we rename /media/volume/MyData/ to /media/volume/MyData-tmp. After attaching the volume, we copy the necessary folders into it.

An important point to note is that the folder /media/volume/MyData/Slicer will not be overwritten if it already exists on the attached volume. This means that if a version of Slicer has already been "installed" into the MyData volume, it will not be updated in subsequent runs unless the existing folder is deleted first.

This behavior is intentional to prevent unintentional overwriting of existing user data.

muratmaga commented 5 days ago

This sounds good.

So if we do want to update, the we can ask users to delete the folder and recreate instance?

jcfr commented 5 days ago

So if we do want to update, the we can ask users to delete the folder and recreate instance?

Exactly.

Alternatively, they could also just locally run the commands that we use in the ansible slicer role (I will document those later after addressing https://github.com/MorphoCloud/MorphoCloudWorkflow/issues/57)

jcfr commented 5 days ago

This has been implemented and deployed across multiple projects:

  1. exosphere via commit https://github.com/MorphoCloud/exosphere/commit/5faa1fbef47fc4fb588b44fa0b577d7d72b64db5 by installing Slicer in MyData volume to persist settings
  2. MorphoCloudWorkflow vis commits https://github.com/MorphoCloud/MorphoCloudWorkflow/compare/0f35a2972567154ac66d9325ca19f72ba0203407...46b028627a49c14294873f5e2a67e08614ab71a6
  3. MorphoCloudInstances via https://github.com/MorphoCloud/MorphoCloudInstances/commit/f256423c631b55b12d3ae3e4a30d0646566dcb5d
muratmaga commented 5 days ago

We still need to increase the disk size to 100GB.

muratmaga commented 4 days ago

working.

muratmaga commented 3 days ago

@jcfr (this is a fresh instance, 76) Looks like /media/volume/MyData-Tmp directory seems to persist. 

exouser@instance-76:~$ cd /media/volume/
exouser@instance-76:/media/volume$ ls -ltr
total 8
drwxr-xr-x 2 exouser exouser 4096 Nov 27 22:00 MyData-tmp
drwxr-xr-x 5 exouser exouser 4096 Nov 27 22:00 MyData
exouser@instance-76:/media/volume$
jcfr commented 3 days ago

re: MyData-tmp

Ditto. It was purposefully not removed. That said, I understand it can lead to confusion for our users.

I will make sure it is removed (while ensuring it is empty before doing so)

jcfr commented 3 days ago

Looks like /media/volume/MyData-Tmp directory seems to persist.

This has been fixed and deployed across multiple projects:

  1. MorphoCloudWorkflow via commit https://github.com/MorphoCloud/MorphoCloudWorkflow/commit/3a43c4d471c4763a0b2548fbae3c3d1822d6b3df
  2. MorphoCloudInstances via https://github.com/MorphoCloud/MorphoCloudInstances/commit/c7fa06bae505c6e024d43fd4543a7b4ae107e455

Listing /media/volume directories confirms that the directory MyData-tmp was effectively removed:

exouser@morpho-cloud-portal-instance-104:~$ ls /media/volume
MyData