feat: Add training pipeline life-cycle management (2/2)

[jsakv]

🚀 This PR introduces the second half of the modifications required to orchestrate and automate our training pipelines. The PR resolve issue #51 :

• Add DVC configurations files and declare training pipeline DVC dags
• Setup a remote artifacts registry/store (GDRIVE)
• Add a CML report comparing the performance of the PR branch to master (see below)

Bonus:

• DVC is compatible with all the cloud providers that offer S3 compatible API (OVH, Scaleway), so we will be able to migrate the artifacts registry by updating our configuration files!

[codecov[bot]]

Codecov Report

Merging #56 (4a1554c) into master (7752c5a) will increase coverage by 0.01%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #56      +/-   ##
==========================================
+ Coverage   95.92%   95.93%   +0.01%     
==========================================
  Files          27       27              
  Lines        1055     1058       +3     
==========================================
+ Hits         1012     1015       +3     
  Misses         43       43              

Flag	Coverage Δ
unittests	95.93% <100.00%> (+0.01%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pyro_risks/config/datasets.py	100.00% <100.00%> (ø)
pyro_risks/pipeline/evaluate.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 7752c5a...4a1554c. Read the comment docs.

[GHCamille]

Thanks for the PR 😍 Quick question : how do we handle this google drive, do we need personal credentials ? how do we access it ? I tried clicking on the URL but no chance so far

[jsakv]

Thanks for the PR 😍 Quick question : how do we handle this google drive, do we need personal credentials ? how do we access it ? I tried clicking on the URL but no chance so far

Thanks, @GHCamille. The google drive is handled via a distinct Gmail address with a

• Specific GCP project
• A service account for allowing DVC push and pull from GitHub CI/CD
• It's all free, and in any case, we have 300€ credits 😄

The files on the drive can be accessed directly, but DVC hashes them. The process for accessing the files and auditing them once the PR will be merge will be the following:

• Checking out the master

• Running dvc pull for setting up credentials

  
  Go to the following link in your browser:
  
  https://accounts.google.com/o/oauth2/auth # ... copy this link

Enter verification code: # <- enter resulting code

- Downloading the oauth2 credentials and storing them outside of the repository ⚠️ 
- Declaring Credentials with 
```sh 
dvc remote modify artifacts-registry gdrive_service_account_json_file_path path/to/file.json

• Running dvc pull again

Since the goal is to deploy the model to production continuously (see below) so we shouldn't directly push the dataset on the drive and should not use dvc push and dvc updatecommands 🚨

[GHCamille]

Works like a charm, thanks. Let's just add here, as a record to first type : dvc remote add -d registry gdrive://1fyD6xyuWWhyjNPoCiCazKshp0yJ8xKs9 dvc pull -r registry In order to create credentials