[Ray CORE] Gitlab integration with authentication

[githuberj]

Description

We would like to utilize gitlab as our collaboration platform. We already utilize KubeRay and would like to enable users to execute RayJobs and RayServings in a secure fashion similar to the one provided for github in https://docs.ray.io/en/latest/ray-core/runtime_env_auth.html. We use a GitOps based approach and therefore would greatly appreciate this possibility.

I currently got it running through an initcotainer with SSH git-sync but I would greatly appreciate a nicer workflow.

I also found a few related issues:

• https://gitlab.com/gitlab-org/gitlab/-/issues/364194 (currently only a Personal Access Token can download the code)
• Some production environments might not have gitlab API access (only ssh based access methods)
• Can't update an application installed from gitlab

Use case

A user of Kuberay can provide a valid authentication method (Personal access token, Group access token, Project access token, SSH key) and the operator and ray cluster can execute his code.

[jjyao]

Does the netrc approach work for gitpod?

[githuberj]

No unfortunately I didn't get gitlab to work with netrc.

[jjyao]

You can check out https://gitlab.com/gitlab-org/gitlab/-/issues/350582, seems there is a solution there

[githuberj]

I tried it but couldn't get it to work with kuberay so I went a step back and tried it on my local:

This code works

import ray

good_env = {"working_dir": "https://github.com/ray-project/serve_config_examples/archive/refs/heads/master.zip"}

# Specify a runtime environment for the entire Ray job
ray.init(runtime_env=good_env,logging_level="DEBUG")

# Create a Ray task, which inherits the above runtime env.
@ray.remote
def f():
    # The function will have its working directory changed to its node's
    # local copy of /tmp/runtime_env_working_dir.
    return open("text_ml.py").read()

print(ray.get(f.remote()))

ray.shutdown()

But if I put the same code into a password protected gitlab repo I only get a RayTaskError(FileNotFoundError): ray::f() (pid=15935, ip=172.26.113.45) exception.

There is also no indication of something going wrong with the runtime environment in the logs. I tried setting NETRC environment variable and using the default location of ~/.netrc

[jjyao]

Make sure you do chmod 600 ~/.netrc. Also can you try wget https://gitlab.com/xxx.zip first to make sure your netrc file is correct.