use input from workflow.benchmark.preparation

RMI-PACTA / workflow.prepare.pacta.indices

This repository is used to run indices through PACTA, and prepare them for transition monitor.

Other

2 stars 0 forks source link

use input from workflow.benchmark.preparation #102

Closed cjyetman closed 3 months ago

cjyetman commented 4 months ago

@jdhoffa I feel like maybe PACTA_DATA_PATH, INDICES_PREPARATION_OUTPUTS_PATH, and the new BENCHMARK_DATA_FILEPATH parameters should be in the config file instead of a .env file? But I tried to follow the existing pattern.

cjyetman commented 4 months ago

note to self: need to update the Readme

AlexAxthelm commented 4 months ago

This PR moves scraping to a different workflow?

cjyetman commented 4 months ago

Yes, that's the idea

AlexAxthelm commented 4 months ago

@jdhoffa I feel like maybe PACTA_DATA_PATH, INDICES_PREPARATION_OUTPUTS_PATH, and the new BENCHMARK_DATA_FILEPATH parameters should be in the config file instead of a .env file? But I tried to follow the existing pattern.

Keeping these as envvars makes sense, in that it introduces a separation between "config" (defining the environment in which the code is run) and "parameters" (defining the specifics of how to run the code). Relevant short read: https://12factor.net/config

jdhoffa commented 3 months ago

Flagging that as of 7ed79f1 , this PR is complete for any LOCAL run of index preparation.

I will begin wiring up the Azure deployment now.

jdhoffa commented 3 months ago

Don't merge until https://github.com/RMI-PACTA/workflow.benchmark.preparation/pull/5 is merged AND runs on Azure.

After this:

[x] Update the config.yml to point to the freshly prepared benchmarks WITH the real MSCI data

cjyetman commented 3 months ago

Don't merge until https://github.com/RMI-PACTA/workflow.benchmark.preparation/pull/5 is merged AND runs on Azure.

Merged

jdhoffa commented 3 months ago

Success!! I think this is ready for review then :-)

jdhoffa commented 3 months ago

@cjyetman I can't ask you to review this since you opened the PR, but would you mind giving it a once-over anyway?

cjyetman commented 3 months ago

@cjyetman I can't ask you to review this since you opened the PR, but would you mind giving it a once-over anyway?

How can I verify if it "worked", beyond no Actions failing?

jdhoffa commented 3 months ago

In this run: https://github.com/RMI-PACTA/workflow.prepare.pacta.indices/actions/runs/9416002748/job/25938190002?pr=102

under "Run container", you can inspect the logs and see that the "real" MSCI indexes are prepared without error: e.g.

Beyond that, you can inspect the outputs of the latest run here: https://portal.azure.com/#view/Microsoft_Azure_FileStorage/FileShareMenuBlade/~/browse/storageAccountId/%2Fsubscriptions%2Ffeef729b-4584-44af-a0f9-4827075512f9%2FresourceGroups%2FRMI-SP-PACTA-DEV%2Fproviders%2FMicrosoft.Storage%2FstorageAccounts%2Fpactadatadev/path/workflow-prepare-pacta-indices-outputs/protocol/SMB

jdhoffa commented 3 months ago

The dir output on Azure is 2023Q4_20240607T110842Z.

For your reference, you can easily download the directory using the az cli with the following command from the terminal (make sure az is installed):

az storage copy --source "https://pactadatadev.file.core.windows.net/workflow-prepare-pacta-indices-outputs/2023Q4_20240607T110842Z" --destination PATH/TO/DOWNLOAD --recursive

cjyetman commented 3 months ago

looks legit...

purrr::map(
  .x = list.files("~/Downloads/2023Q4_20240607T110842Z/", full.names = TRUE),
  .f = ~ unique(readRDS(.x)$portfolio_name)
)
#> [[1]]
#> [1] "iShares Global Corp Bond UCITS ETF"
#> 
#> [[2]]
#> [1] "iShares Global Corp Bond UCITS ETF"
#> 
#> [[3]]
#> [1] "iShares Global Corp Bond UCITS ETF"
#> 
#> [[4]]
#> [1] "iShares Core S&P 500 ETF"            "iShares MSCI ACWI ETF"              
#> [3] "iShares MSCI EM UCITS ETF USD (Acc)" "iShares MSCI World ETF"             
#> [5] "MSCI - ACWI"                         "MSCI - EM (EMERGING MARKETS)"       
#> [7] "MSCI - WORLD"                       
#> 
#> [[5]]
#> [1] "iShares Core S&P 500 ETF"            "iShares MSCI ACWI ETF"              
#> [3] "iShares MSCI EM UCITS ETF USD (Acc)" "iShares MSCI World ETF"             
#> [5] "MSCI - ACWI"                         "MSCI - EM (EMERGING MARKETS)"       
#> [7] "MSCI - WORLD"                       
#> 
#> [[6]]
#> [1] "iShares Core S&P 500 ETF"            "iShares MSCI ACWI ETF"              
#> [3] "iShares MSCI EM UCITS ETF USD (Acc)" "iShares MSCI World ETF"             
#> [5] "MSCI - ACWI"                         "MSCI - EM (EMERGING MARKETS)"       
#> [7] "MSCI - WORLD"

jdhoffa commented 3 months ago

I have manually triggered the "Docker build and test" action from workflow.transition.monitor here: https://github.com/RMI-PACTA/workflow.transition.monitor/actions/runs/9418945968

If all goes well, this SHOULD result in an interactive report that contains the real MSCI index files. (Note: no changes are necessary to any of the config files in workflow.transition.monitor since index preparation also happens on the fly when that action is triggered)