mlops-club / vscode-clearml-session-manager

VS Code extension for create remote workstations (sessions) using ClearML.
Apache License 2.0
16 stars 3 forks source link

license GitHub tag (latest by date) Build, Test, Lint, and Publish codecov

vscode-clearml-session-manager

_The easiest way to connect to ClearML Sessions--one of the best remote workstation offerings in the MLOps space. (comparison of ClearML vs others here)_

💬 We're looking for contributors! See the contributing section below.

ClearML is self-hostable without kubernetes and has a free SaaS-hosted plan, meaning you can get a world-class data science development environment for free.

Watch and learn

soundless demo

How this extension fits into a Cloud or on-prem Architecture

image image image

Contributing

Roadmap

Items marked with ✨ are high-impact, and important for our first release

Features:

Exploring sessions

Connecting to sessions

Low priority it may be of interest to support connecting to sessions by starting an "interactive session" via clearml-session --attach <task id> which creates an SSH tunnel to localhost. This is a different way of connecting to sessions, so this is redundant with the above.

Creating sessions

DevOps

Running the extension locally

VS Code makes it really easy to run extensions and try out code changes:

  1. be sure you have NodeJS installed, some had issues because they had the wrong Node version
  2. go to the src/extension.ts file and press F5 to start a debugging session

Getting up to speed on ClearML and writing VS Code Extensions

Here are a few videos with progress updates. Watching these will step you through how we learned about authoring VS Code extensions and how we got to where we are now.

  1. ~30 min - Announcing the hackathon project
  2. ~30 min - How we got the extension to work with the Python interpreteer by forking the vscode-black-formatter extension
  3. ~45 min - Everything we created/learned during the all-nighter hackathon
    • how to hit the ClearML API
    • how to read the ~/clearml.conf file with TypeScript
    • how we decided to hit the ClearML API from TypeScript rather than Python
    • how we got the list items to show up in the sidebar
  4. Pull request: giving ClearML it's own "View Container" i.e. item in the leftmost sidebar. And how we got our icons to show up in all the right places.
  5. ~5 min - How we got VS Code to open a new window SSH'ed into an already-attached-to ClearML session

Running the extension and ClearML remotely

📌 Note: As a first contribution, it'd be great if you submitted a PR to this README if you get stuck during setup.

Step 1 - set up your own machine, i.e. your laptop

  1. Sign up for the free, SaaS-hosted ClearML at app.clearml.ml
  2. Follow the instructions in their UI to make a set of API keys and put them into a local ~/clearml.conf file by running the clearml-init command and pasting them in.
    1. In case you mess up when doing the initialization, you can go to Settings > Workspace > + Create new credentials in the UI to create a key pair. But it should automataically do this for you when you first log in.

Step 2 - set up a remote machine, e.g. a VM in the cloud

If you choose an EC2 instance or Amazon Lightsail (cheaper than EC2), you can do something like this to install and start up the clearml-agent daemon which will allow you to run ClearML Session on it:

First, connect to it via SSH

ssh ec2-user@<public ip of instance>

Then install and start the clearml-agent daemon

sudo su
yum update -y && yum install -y docker docker-compose python3-pip
service docker start

python -m pip install clearml clearml-agent
clearml-agent init  # paste in your API keys (you can also make a new pair for this)

# exapmle daemon command, you can set these however you like
clearml-agent daemon --queue default --docker --cpu-only --log-level debug

Step 3 - start a ClearML Session on your instance

On your laptop, you'd run something like

clearml-session --queue default --docker python:3.9

Note that this will open an SSH tunnel to your instance, so your instance needs to accept incoming traffic on port 22 and probably others like 10022.

If this is an EC2 instance, this means adjusting your security group. Other clouds have a similar concept. They often call this a "firewall".

Step 4 - run the extension locally

Follow steps 1, 2, 3, and 6 below.

Running the extension and ClearML completely locally

⚠️ Disclaimer: while getting this running locally is a good onboarding exercise, the last step currently fails (For Eric, at least) mid-way through creating the session. If you want to pursue troubleshooting this, by all means!

Unfortunately, for the sake of continuing development, the most reliable course is to provision your own VM in the cloud e.g. an EC2 instance, Digital Ocean droplet, Linode server, etc.

⚠️ Disclaimer: expect problems if you try to run this project directly on Windows.

Install the Windows Subsystem for Linux 2 (WSL2) and develop from there if you are running windows.

The free videos in the Environment Setup section of this course walk you through how to do this, as well as most of step [1] below.

  1. install the prerequisites
    1. docker, on MacOS and Windows (including WSL2), get Docker Desktop
    2. docker-compose, e.g. brew install docker-compose
    3. NodeJS, e.g. with brew install nodejs
    4. Python. pyenv is a good way to install Python.
    5. VS Code
  2. clone the repo
  3. install the NodeJS dependencies
    # cd into the cloned repo and install the NodeJS dependencies
    cd ./vscode-clearml-session-manager/
    npm install
  4. generate a set of ClearML API keys, these get placed at ./dev-utils/volumes/opt/clearml/config/clearml.conf
    npm run create-clearml-credentials
  5. start the ClearML server
    npm run start-clearml-server
  6. start the VS Code extension by opening ./src/extension.ts and pressing F5 on your keyboard
  7. The extension should load successfully, but it won't have any sessions. To start a session, run

    # install the clearml-session CLI into a Python virtual environment
    python -m venv ./venv/
    source ./venv/bin/activate
    npm run install-python-deps
    
    # execute the clearml-session CLI to start a session
    npm run start-clearml-session

    This will take some time to run. While it loads, you should be able to visit http://localhost:8080 and visit the DevOps folder in Clearml after logging in with username: test, password: test.

Architecture of ClearML

ClearML Server with MinIO - Service Architecture

graph LR
    subgraph backend_network["Backend Network"]
        redis[("Redis<br>Port: N/A")]
        mongo[("MongoDB<br>Port: N/A")]
        elasticsearch[("Elasticsearch<br>Port: N/A")]
        fileserver[("Fileserver<br>Port: 8081")]
        minio[("MinIO<br>Port: 9000")]
        apiserver[("API Server<br>Port: 8008")]
        async_delete[("Async Delete<br>Port: N/A")]
        agent_services[("Agent Services<br>Port: N/A")]
    end

    subgraph frontend_network["Frontend Network"]
        webserver[("Webserver<br>Port: 80<br>URL: app.clearml.localhost:8080")]
    end

    user -->|HTTP:8080| webserver
    webserver -->|HTTP:8008| apiserver
    apiserver -->|Internal| mongo
    apiserver -->|Internal| redis
    apiserver -->|Internal| elasticsearch
    apiserver -->|HTTP:8081| fileserver
    apiserver -->|HTTP:9000| minio
    fileserver -->|HTTP:9000| minio
    async_delete -->|Internal| apiserver
    agent_services -->|Internal| apiserver

    style webserver fill:#f9f,stroke:#333,stroke-width:2px

Notes

Service URL Notes
UI http://app.clearml.localhost:8080 user: test pass test
API Server http://api.clearml.localhost:8008
Fileserver http://files.clearml.localhost:8081
MinIO UI http://minio.clearml.localhost:9001 user: minioadmin pass: minioadmin

Ports mentioned as "N/A" are not directly exposed to the host machine but are used internally within the Docker network for service communication.

Remember to replace the example URL with the actual domain and port you will use in your production or development environment. The above diagram assumes that services like MongoDB, Redis, and Elasticsearch do not need to be accessed directly through a browser and therefore do not have a URL associated with them for external access.