[GCC] PAD: Gerar Versão CD_CT

[joaomas]

:people_holding_hands: User Story

COMO Saulo, QUEREMOS atualizar o repositório, PARA disponibilizar a versão mais atualizada para a comunidade.

:clipboard: Critérios de aceite de conclusão da Issue

• [x] Criar Release
• [x] Atualizar README
• [x] History
• [x] Garantir que todas variáveis implementadas foram incluídas
• [x] Deixar como default a ultima versão do MONAN-Model (0.5.0)
• [x] Deixar como default a ultima versão do Convert_Mpas (1.0.0)
• [x] Incluir zgeo no streams (Model) e include (Pos)
• [x] Tirar chamada do 5.prods testar o CRON
• [x] Testar na versão do TC (corrigir)
• [x] Tag 0.1.0

:pencil: Detalhamento adicional da atividade

-.

:comet: Impactos

-.

:link: Dependências

• 492

• 489

• 488

• 485

:hammer_and_wrench: Solução

• Release 0.1.0 criado pelo Denis - 9/5/24

• zgeo inserido no streams... e include... pelo Khamis - 9/5/24

A (Primeiros testes)

• Atualizando e igualando as variáveis para as saídas Model e Post.

$ git checkout develop $ git flow init $ git pull $ git flow feature start scripts_CD-CT-493 $ git push --set-upstream origin

• Alterado os arquivos ..datain/namelists/ "include_fields.diag" e "stream_list.atmosphere.diagnostics" inserindo todas as variaveis solicitadas da planilha, totalizando 673
• Criado os arquivos include_fields.diag.CD-CT e stream_list.atmosphere.diagnostics.CD-CT
• Atualizado README.md

$ git push

• Testes na Egeon - rodada 24 horas (enviado e-mail ao Paulo Kubota)
• Analise do time sobre o erro das variáveis.

$ git flow feature finish --keeplocal scripts_CD-CT-493

Switched to branch 'develop' Your branch is up to date with 'origin/develop'. Merge made by the 'recursive' strategy. README.md | 19 ++- datain/namelists/include_fields.diag | 198 ++++++++++++++++++++--- datain/namelists/include_fields.diag.CD-CT | 673 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ datain/namelists/stream_list.atmosphere.diagnostics | 154 ++++++++++++++++++ datain/namelists/stream_list.atmosphere.diagnostics.CD-CT | 673 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 1689 insertions(+), 28 deletions(-) create mode 100644 datain/namelists/include_fields.diag.CD-CT create mode 100644 datain/namelists/stream_list.atmosphere.diagnostics.CD-CT To github.com:monanadmin/scripts_CD-CT.git

• [deleted] feature/scripts_CD-CT-493

Summary of actions:

• The feature branch 'feature/scripts_CD-CT-493' was merged into 'develop'
• Feature branch 'feature/scripts_CD-CT-493' is still locally available; it has been remotely deleted from 'origin'
• You are now on branch 'develop'

$ git checkout feature/scripts_CD-CT-493 $ git branch -m feature/scripts_CD-CT-493 feature/scripts_CD-CT-493_CLOSED $ git push origin -u feature/scripts_CD-CT-493_CLOSED

Teste 10 dias Previsão - 4 nodes

MONAN-Model - Develop scripts_CD-CT - Develop

1 - install real 3m7.199s user 4m9.331s sys 0m18.411s

2 - pré real 22m48.528s user 10m35.475s sys 1m33.751s

3 - run real 374m5.599s user 0m1.263s sys 0m0.554s

4 - pos real 4m10.533s user 0m2.733s sys 0m3.506s

5 - prods real 0m12.107s user 0m1.159s sys 0m0.260s

Saidas Tamanho

Pre - 9.3G Model - 619G Post - 158G Prods - 1M

Total: 786G

B (Default 0.5.0 - Model e 1.0.0 - Convert)

Alterado código para deixar default 0.5.0 (model) e 1.0.0 (convert)

tag_or_branch_name_MONAN=${tag_or_branch_name_MONAN:="0.5.0"}

tag_or_branch_name_CONVERT_MPAS=${tag_or_branch_name_CONVERT_MPAS:="1.0.0"}

C (Alterações para atender o cron)

As adaptações no scripts_CD_CT para funcionar o cron foram:

• Criação do cron_CD-CT.bash (mix do "0.run_all_first_time.bash" e "rmonancron.sh").
• Alteração "1.install_monan.bash" em:

add linha 79 DATAOUTCRON=${DATAOUT}/cron; mkdir -p ${DATAOUTCRON}

add Linha 100 $(sed -i "s;MONANDIR=.*$;MONANDIR=$MONANDIR;" setenv.bash)

• Alteração "setenv.bash" em:

(alt) 31 e 32

export DIR_SCRIPTS=$(dirname $(dirname $(pwd)))
export DIR_DADOS=$(dirname $(dirname $(pwd)))

(add) 33 export MONANDIR=$MONANDIR

//

Removido chamada para o script 5 (prods) no cron.

Removido copia png para FTP e substituído por cópia do cron.log.

//

$ git flow init

Which branch should be used for bringing forth production releases?

• main
• release/0.1.0 Branch name for production releases: [main]

Which branch should be used for integration of the "next release"?

• release/0.1.0 Branch name for "next release" development: [release/0.1.0]

How to name your supporting branch prefixes? Feature branches? [feature/] Bugfix branches? [bugfix/] Release branches? [release/] Hotfix branches? [hotfix/] Support branches? [support/] Version tag prefix? [] Hooks and filters directory? [/home/joao.messias/repo/scripts_CD-CT/.git/hooks]

$ git flow release finish 0.1.0

Switched to branch 'main' Your branch is up to date with 'origin/main'. Merge made by the 'recursive' strategy. .gitignore | 5 + README.md | 41 +++--- datain/namelists/convert_mpas.nml | 8 ++ datain/namelists/include_fields.diag | 702 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- datain/namelists/include_fields.diag.CD-CT | 695 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ datain/namelists/namelist.atmosphere.TEMPLATE | 2 +- datain/namelists/stream_list.atmosphere.diagnostics | 751 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----------- datain/namelists/stream_list.atmosphere.diagnostics.CD-CT | 695 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ datain/namelists/stream_list.atmosphere.output | 4 +- datain/namelists/stream_list.atmosphere.surface | 0 datain/namelists/streams.atmosphere.TEMPLATE | 6 +- datain/namelists/target_domain | 6 + scripts/0.run_all_first_time.bash | 72 +++++------ scripts/1.install_monan.bash | 146 +++++++++++---------- scripts/2.pre_processing.bash | 6 + scripts/3.run_model.bash | 35 ++--- scripts/4.run_post.bash | 316 +++++++++++++++++++++++++++++++++++++++------ scripts/5.run_products.bash | 64 ++++++---- scripts/6.do_manut.bash | 27 ++++ scripts/cron_cd-ct.bash | 74 +++++++++++ scripts/group_levels.py | 93 ++++++++++++++ scripts/make_degrib.bash | 1 + scripts/make_initatmos.bash | 1 + scripts/make_static.bash | 2 +- scripts/setenv.bash | 30 ++++- scripts/setenv_python.bash | 6 + tests/test_group_levels.py | 95 ++++++++++++++ 27 files changed, 3572 insertions(+), 311 deletions(-) create mode 100644 .gitignore create mode 100755 datain/namelists/convert_mpas.nml mode change 100755 => 100644 datain/namelists/include_fields.diag create mode 100644 datain/namelists/include_fields.diag.CD-CT mode change 100755 => 100644 datain/namelists/stream_list.atmosphere.diagnostics create mode 100644 datain/namelists/stream_list.atmosphere.diagnostics.CD-CT mode change 100755 => 100644 datain/namelists/stream_list.atmosphere.output mode change 100755 => 100644 datain/namelists/stream_list.atmosphere.surface create mode 100755 datain/namelists/target_domain create mode 100755 scripts/cron_cd-ct.bash create mode 100644 scripts/group_levels.py create mode 100644 scripts/setenv_python.bash create mode 100644 tests/test_group_levels.py Already on 'main' Your branch is ahead of 'origin/main' by 40 commits. (use "git push" to publish your local commits) Switched to branch 'release/0.1.0' Your branch is up to date with 'origin/release/0.1.0'. Already up to date! Merge made by the 'recursive' strategy. Switched to branch 'main' Your branch is ahead of 'origin/main' by 40 commits. (use "git push" to publish your local commits) To github.com:monanadmin/scripts_CD-CT.git

• [deleted] release/0.1.0 error: The branch 'release/0.1.0' is not fully merged. If you are sure you want to delete it, run 'git branch -D release/0.1.0'.

Summary of actions:

• Release branch 'release/0.1.0' has been merged into 'main'
• The release was tagged '0.1.0'
• Release tag '0.1.0' has been back-merged into 'release/0.1.0'
• Release branch 'release/0.1.0' is still locally available; it has been remotely deleted from 'origin'
• You are now on branch 'main'

$ git checkout main
$ git push
$ git checkout develop
$ git merge main
$ git push

//

Nota 1: Por algum motivo a develop não foi atualizada. Foi necessário fazer merge main e depois subi-la. Nota 2: O release remoto foi excluído. Foi necessário fazer.

git checkout release/0.1.0
git branch --unset-upstream
git push --set-upstream origin release/0.1.0

//

Tag 0.1.0 gerada: https://github.com/monanadmin/scripts_CD-CT/releases/tag/0.1.0

:rotating_light: Problemas encontrados

• Aguardando issues dependentes.
• Foi necessário aumentar o wall_time em setenv para 8 horas para o teste de 10 dias.
• Script 4-Pós aparentemente finaliza sem aguardar o fim da submissão.
• Script 5-Prods a legenda aparece a data default e não a parametrizada e a tag 0.4.0 (linhas 103 e 109).

draw title Previsao de Precipitacao \ MONAN: prod: 2024010100 valid: 2024010102' (...) 'draw string 0.1 0.1 MONAN v.0.4.0'

:white_check_mark: Conclusão

• Atualizado e igualada todas as variáveis para serem processadas no Model e Post com a feature "scripts_CD-CT-493_CLOSED".
• Testes com 10 dias usando a develop do CD-CT e develop do MONAN-Model realizado.
• Melhorias nos scritps: a) Legenda em PRODS do png aponta para o dia da data default do README. b) No Post ele finaliza mais não aguarda o fim das Submissões. (poucos minutos depois).
• Inserido o default 0.5.0 - Model e 1.0.0 do Convert em install.
• Add cron_CD-CT.bash em scripts para atender ao cron.
• Criado variável global $MONANDIR para armazenar a última instalação.
• Testes finais apontaram erro em zgeo e umidade relativa que serão tratadas em outra issue.
• Tag 0.1.0 gerada.

:spiral_calendar: Trabalhos Futuros

-.

[joaomas]

Variáveis incluídas na Sprint 61:

rucuten - 462 rvcuten e rvblten - 463 cldfrac - 464 rublten - 486 rthratensw - 487 omega - 465

[joaomas]

- Criar Feature para trabalhar

- Arquivos para alterarem

• Pós scripts_CD-CT/datain/namelists/include_fields.diag

• Model scripts_CD-CT/datain/namelists/stream_list.atmosphere.diagnostics

- Sugestão e Alerta do CR

• A configuração do Model com o CD_CT devem caminhar juntos, caso o usuário tenha um Model (Fork) sem o Registry atualizado com as variáveis solicitadas no stream_list dará erro. Sugestão criar um default básico do stream e criar um include_fields.diag.CD_CT e stream_list.atmosphere.diagnostics.CD_CT específicos para o CD_CT.

[joaomas]

@marcelopaivaramos por favor, faça um teste do procedimento da DTN06 (pagina 13 em diante) em um repositório de teste da sua conta no github.

[joaomas]

Add Streams e Input

zgeo_15hPa zgeo_20hPa zgeo_30hPa zgeo_50hPa zgeo_70hPa zgeo_100hPa zgeo_150hPa zgeo_200hPa zgeo_250hPa zgeo_300hPa zgeo_400hPa zgeo_500hPa zgeo_600hPa zgeo_700hPa zgeo_825hPa zgeo_850hPa zgeo_875hPa zgeo_900hPa zgeo_925hPa zgeo_950hPa zgeo_975hPa zgeo_1000hPa

[joaomas]

Editar

.setenv

export MODEL_QUEUE=batch export MODEL_ncores=1024 export MODEL_nnodes=16 export MODEL_ncpn=64 export MODEL_jobname="Model.MONAN" export MODEL_walltime="4:00:00"

[joaomas]

Teste rápido realizado no egeon com a conta monan com sucesso.