Expose a variable where each dependencies are cloned to

[marcm-ml]

Is your feature request related to a problem? I want to dynamically access the directory from an imported devspace.yaml file from another repository. As I can see, in imports the repositories are cloned into ~/.devspace/dependencies/<dynamically-created-dir>. I want to know in advance, what the <dynamically-created-dir> is, such that I can access certain files from that repository (scripts in my case).

Which solution do you suggest? I suggest to expose a variable such that these dynamically created dirs are properly accessible. Alternatively, I suggest to include another field in imports (for git repos) which can overwrite the <dynamically-create-dir> to a static one provided by the user through devspace.yaml.

Which alternative solutions exist? None

Additional context Ask if needed, please.

Code Example

version: v2beta1
name: devspace.yaml

# Import a template
# Depending on the URL and branch, tag or revision, the resulting path is different,
# which makes it difficult to "guess" the path in advance.
imports:
  - git: <URL-to-repo1>
    branch: main

vars:
  # this could be a possible runtime variable exposed
  # would resolve to: ~/.devspace/dependecies/<URL-to-repo1>-<revision>/<branch>/<tag>
  IMPORT_DIR_REPO1: ${runtime.imports.clonepath}
``

[89luca89]

Hi @marcm-ml thanks for opening this issue, will look into this :+1:

[marcm-ml]

Hey @lizardruss I saw your comment as an email and I think all of my issues could be solved if each devspace.yaml would be aware in which directory it is lying (not executed). For example, this could be a variable similar to DEVSPACE_NAMESPACE. If the devspace.yaml can then access this variable, we can just have relative imports based upon this. I think making this an internal variable would be quite easy as this could either be the path to the devspace.yaml where devspace dev is executed in or the cloned git path (available from getDependencID here) or the local path to an imported dependency (also easy to get from the underlying yaml file and its parent path)

However, this variable cannot be the same for my main project (located in ".") and the imported dependency ("~/.devspace/...") otherwise it is the same issue, e.g. we cannot have relative imports in the imported dependency. This could be achieved with runtimes variabel or by internally overwriting the variable at import time similar to how we can overwrite vars for imported dependencies? But this might be trickier.

I can provide a more straight forward example of my usecase(s) later, if interested.

[lizardruss]

@marcm-ml Yeah, I'm trying to arrive at a solution that won't be problematic. Do you need this for both dependencies and imports or only imports?

Originally I was thinking runtime variables of the form ${runtime.import[0].path} or ${runtime.dependency.dep-name.path} could be used. The limitation is that runtime variables are not available in pipelines, which might limit the usefulness. Referencing imports is also awkward, since its an array. It should be possible to add a pipeline function to read the runtime vars.. but will discuss with my colleagues before going that route.

Then I was thinking we could populate vars from runtime variables, like your example, but that's not currently possible due to config load sequencing. There's a similar issue with the built-in ${DEVSPACE_*} style variables, since their definition is static and needs to happen before loading dependencies occurs, IIRC.

Next, the question you saw (but then I deleted).. would it be better to have these scripts encapsulated by an exported function? Then I realized that imports just get copied into the current config, so any paths would be relative to the current project at execution time, which isn't what you want. It would also be a large undertaking to get imported functions to work that way.

imports also allow importing variables and resolving naming conflicts, but I'm not sure that it would work for a built-in variable like ${DEVSPACE_CONFIG_PATH}, if that were to be added. It's really only intended to import user defined vars. I'll try it out just to be sure. I don't think modifying imports to work with built-in variables would be good, since you'd need to always rename them otherwise really odd things might happen.

So with all that said.. getting the path where the import / dependency is stored is the easy part, but making a coherent way to reference it, and have it work consistently in different parts of the DevSpace schema is pretty hard! Your alternative suggestion of adding an option to specify where the dependency should get stored would likely be the easiest choice. An example of what that might look like:

version: v2beta1
name: devspace.yaml

vars:
  IMPORT_1_CLONE_PATH: ~/.dependencies/my-special-import-path

# Import a template
# Depending on the URL and branch, tag or revision, the resulting path is different,
# which makes it difficult to "guess" the path in advance.
imports:
  - git: <URL-to-repo1>
    branch: main
    clonePath: ${IMPORT_1_CLONE_PATH}

This keeps things simple since you'd be able to use that anywhere vars are available, but puts deciding the paths on you. I'd also need to test that the devspace reset dependencies command also works correctly if this is the solution. I'll keep testing things out to see if there's something more convenient. This is the most appealing option in terms of complexity, so I'll probably proceed with this unless you're opposed.

One more suggestion might be to inline your scripts into functions in the imported devspace.yaml. That way you won't have to think about paths at all but I don't think that will always be practical.

[marcm-ml]

Hey @lizardruss, what is the progress on the PR draft? I would really love this feature and it would make our workflow much simpler

[mrkwtz]

We have the same "problem". Would love to just reference a bash file inside the imported devspace path. That way we don't need to "wrap" the content of these scripts into devspace functions.

[lizardruss]

@marcm-ml @mrkwtz Would you two be willing to try out or at least review this PR?

It's been a little bit since I've looked into this issue, but I remember being uncertain whether the solution in the PR would actually meet your needs. Feedback would be great, but I think it still needs review from our product team to move forward too.

[marcm-ml]

Hey I tried the MR and this would solve my problems. We use a central template which contains some script, configs, etc. We want to copy them into the pod at runtime. Albeit we now solved it differently using prebuilt docker images. However, it would still be a very appreciated feature and would make some of our future usecases possible which we have already worked out but are just waiting for this MR to be merged and released into the cli.

Cheers

[mrkwtz]

Hey @lizardruss, thanks for your comments / PR. I guess it could somehow solve our "problem" but it's not very user friendly. I'll try to describe our use-case and why.

So let's assume we have two repositories: one that includes scripts for automatically getting database credentials (from here on called "db-creds") and one that hosts a ML microservice (from here on called "ml-service). Both have a devspace.yaml. For the ml-service for local development developers need to obtain database credentials to pull and push data there.

Right now what they need to do is: clone the db-creds repo, execute the scripts that obtain these credentials, clone the ml-service repo. To improve that workflow it would be nice if we could just import the db-creds repo via DevSpace inside the ml-service devspace.yaml. So the devspace.yaml inside the db-creds repo could define functions like "get-credentials" which could look like

functions:
  get-credentials: |-
    ./db-creds.sh "some" "args"

if some args or the name of the script changes, the services (like ml-service) that use that import would not need to be bothered and just call "get-credentials". With your solution, the repo that uses the db-creds functions, would need to know the name of the script etc. It would be nice if DevSpace would be aware of the contexts for both the imported devspace.yaml and the own one.

So if the ml-service devspace.yaml calls a "get-credentials" function like above DevSpace knows that it's imported and is also aware of the directory context of that import if the function is executed (i.e. it knows that it has to look in the imported folder for the db-creds.sh).

I hope I could explain our use-case a little bit better. If you have any questions I'm more than happy to answer them :) That feature would allow us to re-purpose so much of our already existing tooling (scripts) via DevSpace, it would be awesome.

[marcm-ml]

Ok i could finally give it a chance to build a binary from the MR and test it with out setup. Unfortunately, the way it is implemented right now is not sufficient. I can get the runtime variables via the get_runtime_variable function in the pipelines. However, I cannot access it via ${runtime.XXX} in the devspace.yaml which would really be the use case we would need. Am i missing something? I believed I could simply do something like that:

vars:
  PATH: ${runtime.imports[0].path}

but that does not work.

[marcm-ml]

Specific use case:

Some of our dependencies have their own helm chart defined in the helm directory. Currently the way we have implemented the deployment of these dependencies is that we do something along the lines of: Project'a A devspace.yaml:

vars:
  SYNC_DIR: "."
deployments:
  application:
    helm:
      name: ${SYNC_DIR}/helm

To make the above work when using Project A as a dependencies for project B, SYNC_DIR would not resolve properly and we override SYNC_DIR specifically for project B to point to the local dir of project A. However, when using git this does not really work as we have no way of knowing the clone dir in advance

[marcm-ml]

Okay, I think what I mentioned above can be partially resolved with dependencies. As with many things, it came down to us mixing up imports and dependencies. When using imports the relative paths are obviously resolved incorrectly when assuming that what you import defines something that is specific to project-b but not true for project-a (such as helm chart location). This applies to the example above, where we used imports for using dependency B in project A. This is however not working as mentioned above.

However, if you do the same via dependencies (e.g. define project-b as dependency in project-a) then the helm upgrade command is correctly executed inside the folder where project-b will be cloned to or is referenced.

This make sense if you also read the section in the docs:

Difference	imports	dependencies
Use Case	Importing shared/standardized functionality (e.g. functions, pipelines, commands) into independent projects	Defining logical dependencies between inter-dependentprojects (app A requiring app B to be deployed)

E.g. Imports are used when you want to share specific things such as a deployment method which is true regardless of the project. Dependencies should be correctly used in cases where A is dependent on B.

Nevertheless, i think having the option to get the path of some imported config is very useful as mentioned by others in this issue

[lizardruss]

@marcm-ml Glad you were able to get it working with dependencies!

@mrkwtz I wonder if your use case could be addressed by defining a custom pipeline get-credentials that executes the required script in db-creds, and then including that as a dependency and executing it with run_dependency_pipelines --pipeline get-credentials. This essentially turns a dependency pipeline into an importable function.

The main issue with getting this to work for imports is that imports do not have an execution context, and instead are intended only as a way to deduplicate repeat configuration. To get them to have their own execution context would be quite a bit of refactoring. Let us know if the dependency-pipeline-as-a-function would be an acceptable solution.

Thanks!

[karbowiak]

Any chance this can be somewhat expanded, to have the path that an import got cloned to? 😅

Currently i have to copy a file from the repo that got cloned into position, but i end up doing..

cp ~/.devspace/dependencies/whatever-the-repo-name-is/... to/path

Would be awesome if i could just do

cp ${runtime.imports[0].cloned-to-path}/... to/path

Or something like it 😅

(I did check the pull request, but don't see anything in regards to the cloned path, only the import path)

[lizardruss]

@karbowiak runtime.dependencies.[name].path from the PR should be the path the dependency was cloned to.

I still wonder though if using a pipeline defined by the dependency would be a better path. I may put together an example to try it myself.