search

JCSDA / spack-stack

Creative Commons Zero v1.0 Universal
27 stars 47 forks source link

Soft links to latest modules on each platform #1204

Closed AlexanderRichert-NOAA closed 1 month ago

AlexanderRichert-NOAA commented 4 months ago

The idea was raised within EMC to create a symlink, something like /path/to/spack-stack/latest-modules, on each platform that would point to the most recent $env/install/modulefiles/Core directory. This would be useful for developers who don't mind updating a bunch of package versions at once, and keeps them from having to update their application modulefiles, at least on develop branches. Let's try this on one or two systems to make sure it works, then it should be as simple as reassigning the symlink with each release.

spack-stack-1.8.0:

AlexanderRichert-NOAA commented 3 months ago

@climbfuji @RatkoVasic-NOAA please check out the paths in the description and add symlinks on any other systems where they might get used. These are all pointing to 1.7.0 (I didn't do Gaea C6 since we didn't deploy 1.7.0 there)

module use /ncrc/proj/epic/spack-stack/latest-ue-intel && module load stack-intel && module load ..., e.g., seems to work on all of the systems in the description based on loading a few modules, which'ing a few executables, etc.

I propose we create an issue with each release to remind ourselves to update these symlinks, and try to update them on the same day across systems to avoid excessive confusion and heartache for developers.

Relatedly, I created a wiki page to document all currently supported spack-stack environments on each platform: https://github.com/JCSDA/spack-stack/wiki/Extended-list-of-release-directories If it looks good, we can coordinate dividing up the work, which I predict will be slightly tedious but shouldn't take too long.

climbfuji commented 3 months ago

Thanks @AlexanderRichert-NOAA . We should focus on the NOAA/EPIC platforms for the symlinks first - suggest doing S4 next.

RatkoVasic-NOAA commented 1 month ago

@AlexanderRichert-NOAA I created symbolic links to the latest spack-stack (1.8.0) on 10 tier1 machines.

climbfuji commented 1 month ago

@AlexanderRichert-NOAA I created symbolic links to the latest spack-stack (1.8.0) on 10 tier1 machines.

  • We have to think of some kind of reminder, task, TODO or something that will remind us on updating it with every new release.
  • Also, this list should be available somewhere for people who need it.

How about you add it to the already closed spack-stack-1.8.0 release task issue (with the list of platforms and a reference to this issue for the location)? I usually take the last releases' task issue to create the next one.

climbfuji commented 1 month ago

Actually, let me do this real quick.

climbfuji commented 1 month ago

Done. See https://github.com/JCSDA/spack-stack/issues/1278#issue-2494835661

RatkoVasic-NOAA commented 1 month ago

That was quick

climbfuji commented 1 month ago

I see S4 is missing on this list. Is that needed?

RatkoVasic-NOAA commented 1 month ago

I don't have access there

climbfuji commented 1 month ago

That's why I am asking - JCSDA (@srherbener) or I can do that if it's required

srherbener commented 1 month ago

These directories might work for the S4 symlinks

/data/prod/contrib/spack-stack /data/prod/jedi/spack-stack

I vote for /data/prod/jedi/spack-stack and if that is agreeable, I can create the links. (I don't have write permission for the /data/prod/contrib directory.

climbfuji commented 1 month ago

These directories might work for the S4 symlinks

/data/prod/contrib/spack-stack /data/prod/jedi/spack-stack

I vote for /data/prod/jedi/spack-stack and if that is agreeable, I can create the links. (I don't have write permission for the /data/prod/contrib directory.

Makes sense to me, thanks @srherbener. This is also the top-level spack-stack directory, and therefore matches what is done on the other NOAA platforms.

AlexanderRichert-NOAA commented 1 month ago

Let's close this once Steve's done with S4 and after #1335 is merged.

srherbener commented 1 month ago

S4 has a symlink for latest-ue-intel now. For a while, Intel is the only compiler we've been supporting on S4.

[jedipara@s4-submit ~]$ ll /data/prod/jedi/spack-stack/
total 197
drwxr-xr-x   8 jedipara domain users 11776 Aug 26  2022 ecflow-5.8.4
lrwxrwxrwx   1 jedipara domain users    66 Oct  9 20:08 latest-ue-intel -> spack-stack-1.8.0/envs/ue-intel-2021.10.0/install/modulefiles/Core
drwxr-xr-x  18 jedipara domain users 12800 Aug 26  2022 miniconda-3.9.12
drwxr-xr-x   7 jedipara domain users 11776 May 28  2023 modulefiles
drwxr-xr-x  10 jedipara domain users 12800 Feb 22  2023 mysql-8.0.31
drwxr-sr-x 375 jedipara domain users 57856 Oct  7 03:05 source-cache
drwxr-xr-x   9 jedipara domain users 12800 May 28  2023 spack-stack-1.4.0
drwxr-xr-x   9 jedipara domain users 12800 Aug 14  2023 spack-stack-1.4.1
drwxr-xr-x  10 jedipara domain users 13824 Oct 12  2023 spack-stack-1.5.0
drwxr-xr-x  10 jedipara domain users 12800 Nov 17  2023 spack-stack-1.5.1
drwxr-xr-x  10 jedipara domain users 13824 Jul  8 14:59 spack-stack-1.6.0
drwxr-xr-x  11 jedipara domain users 13824 May 23 22:40 spack-stack-1.7.0
drwxr-xr-x  11 jedipara domain users 12800 Oct  7 03:05 spack-stack-1.8.0
climbfuji commented 1 month ago

Thanks @srherbener! I updated the description of the issue (first comment) with the link on S4.