How to integrate applications with package hierarchy?

[SanderMertens]

Currently, corto stores applications in {environment}/bin/cortobin/{corto version}/application name and packages in {environment}/lib/corto/{corto version}/package name. This causes a few issues:

• Applications do not show up in the virtual store when doing a corto_select, because it only checks {environment}/lib
• Applications cannot be created with nested identifiers foo/bar/helloworld without having foo/bar as a package
• When creating applications with the same name as a package, this can create conflicts with files that are installed to {environment}/etc and {environment}/include
• The cortobin prefix for binaries is a bit quirky- but needed because otherwise corto would conflict with the corto tool that is installed to {environment}/bin.
• Because corto_locate only searches {environment}/lib, it is impossible to resolve etc and include folders for applications.

Note that corto applications are run with corto run, as users will not likely directly invoke the application from its actual location, like:

$ ~/.corto/bin/cortobin/1.3/foo/bar/bar

as opposed to:

$ corto run foo/bar

This means that if packages/binaries are going to be relocated, this can be done in a way that is transparent for users.

A solution needs to be devised where:

• applications show up in the virtual store
• there can be no name clashes between applications and packages
• corto_locate is able to find applications + their etc and include folders
• applications can be created with nested identifiers, just like packages

It seems unlikely that corto will be able to support packages/applications with the same name going forward, so assuming this is the case, there are a few possible solutions:

1. Leave the current structure as is, but take bin path into consideration by corto_locate and the loader mount

pros:

• libraries and binaries are properly separated in bin and lib

cons:

• corto now has to search twice as many paths, which can result in measurable slowdowns when iterating the object store and building projects
  • the corto loader mount has to interleave results from 4 (and at worst 6) paths. This happens in every corto_select, unless automatic package loading has been turned off.
  • harder to debug as package errors can only be solved by both looking at bin and lib (this is not an issue today as apps are not taken into consideration at all by corto so they can't cause issues)

2. Store both applications and packages in lib

pros:

• the {environment}/lib directory structure exactly mirrors the package hierarchy (simpler to debug)
• all existing functionality for locating/iterating packages automatically works for applications as well
• getting rid of the quirky cortobin path which simplifies code in a number of locations
• no performance hit
• easier to remove all corto files from a system

cons:

• executables in the lib path. This is arguably not an issue since the user will never see the actual location of a binary anyway

3. Store applications and packages in /opt/corto

pros:

• no binaries in lib
• no performance hit

cons:

• complicates the installer, as the corto environment can no longer be simply copied from ~ to /usr/local
• include files still have to be stored in /usr/local (etc may or may not be) which spreads files across more locations on the filesystem
• on some operating systems it is possible to write to /usr/local without asking for permission, whereas this is not possible with /opt (usability issue)

It's obvious which solution I'm leaning towards- but look forward to hearing more pros/cons/approaches.

[SanderMertens]

After discussion on Slack, it was proposed to store both packages and applications in lib, optionally with creating softlinks to bin for applications.