search

con / duct

A helper to run a command, capture stdout/stderr and details about running
MIT License
1 stars 1 forks source link

incomplete output of stdout is stored (when process died) #27

Closed yarikoptic closed 1 month ago

yarikoptic commented 1 month ago
I had the process running which produced following output on the screen ```shell (git)smaug:~/proj/repronim/containers[master]git $> datalad run -m "Update neurodesk image urls" duct scripts/replace_neurodesk_urls [INFO ] == Command start (output follows) ===== INFO: file images/neurodesk/neurodesk-afni--21.2.00.simg OK: https://d15yxasja65rk8.cloudfront.net/afni_21.2.00_20210714.simg INFO: adding https://d15yxasja65rk8.cloudfront.net/afni_21.2.00_20210714.simg to images/neurodesk/neurodesk-afni--21.2.00.simg ^[[0Jaddurl https://d15yxasja65rk8.cloudfront.net/afni_21.2.00_20210714.simg ok ^[[1G INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--21.2.00.simg ok rmurl images/neurodesk/neurodesk-afni--21.2.00.simg ok rmurl images/neurodesk/neurodesk-afni--21.2.00.simg ok INFO: file images/neurodesk/neurodesk-afni--22.1.14.simg ERROR: https://d15yxasja65rk8.cloudfront.net/afni_22.1.14_20220713.simg - could not verify presence: 403 . Will not be added INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--22.1.14.simg ok rmurl images/neurodesk/neurodesk-afni--22.1.14.simg ok rmurl images/neurodesk/neurodesk-afni--22.1.14.simg ok INFO: file images/neurodesk/neurodesk-afni--22.3.06.simg INFO: removing 3 oracle urls ERROR: https://d15yxasja65rk8.cloudfront.net/afni_22.3.06_20221128.simg - could not verify presence: 403 . Will not be added rmurl images/neurodesk/neurodesk-afni--22.3.06.simg ok rmurl images/neurodesk/neurodesk-afni--22.3.06.simg ok rmurl images/neurodesk/neurodesk-afni--22.3.06.simg ok INFO: file images/neurodesk/neurodesk-afni--22.3.07.simg INFO: adding https://d15yxasja65rk8.cloudfront.net/afni_22.3.OK: https://d15yxasja65rk8.cloudfront.net/afni_22.3.07_20221206.simg 07_20221206.simg to images/neurodesk/neurodesk-afni--22.3.07.simg ^[[0Jaddurl https://d15yxasja65rk8.cloudfront.net/afni_22.3.07_20221206.simg ok ^[[1G INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--22.3.07.simg ok rmurl images/neurodesk/neurodesk-afni--22.3.07.simg ok rmurl images/neurodesk/neurodesk-afni--22.3.07.simg ok INFO: file images/neurodesk/neurodesk-afni--23.0.00.simg INFO: removing 3 oracle urls ERROR: https://d15yxasja65rk8.cloudfront.net/afni_23.0.00_20230118.simg - could not verify presence: 403 . Will not be added rmurl images/neurodesk/neurodesk-afni--23.0.00.simg ok rmurl images/neurodesk/neurodesk-afni--23.0.00.simg ok rmurl images/neurodesk/neurodesk-afni--23.0.00.simg ok INFO: file images/neurodesk/neurodesk-afni--23.0.04.simg ERROR: https://d15yxasja65rk8.cloudfront.net/afni_23.0.04_20230215.simg - could not verify presence: 403 . Will not be added INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--23.0.04.simg ok rmurl images/neurodesk/neurodesk-afni--23.0.04.simg ok rmurl images/neurodesk/neurodesk-afni--23.0.04.simg ok INFO: file images/neurodesk/neurodesk-afni--23.0.07.simg OK: https://d15yxasja65rk8.cloudfront.net/afni_23.0.07_20230302.simg INFO: adding https://d15yxasja65rk8.cloudfront.net/afni_23.0.07_20230302.simg to images/neurodesk/neurodesk-afni--23.0.07.simg ^[[0Jaddurl https://d15yxasja65rk8.cloudfront.net/afni_23.0.07_20230302.simg ok ^[[1G INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--23.0.07.simg ok rmurl images/neurodesk/neurodesk-afni--23.0.07.simg ok rmurl images/neurodesk/neurodesk-afni--23.0.07.simg ok INFO: file images/neurodesk/neurodesk-afni--23.3.02.simg OK: https://d15yxasja65rk8.cloudfront.net/afni_23.3.02_20231024.simg INFO: adding https://d15yxasja65rk8.cloudfront.net/afni_23.3.02_20231024.simg to images/neurodesk/neurodesk-afni--23.3.02.simg ^[[0Jaddurl https://d15yxasja65rk8.cloudfront.net/afni_23.3.02_20231024.simg ok ^[[1G INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--23.3.02.simg ok rmurl images/neurodesk/neurodesk-afni--23.3.02.simg ok rmurl images/neurodesk/neurodesk-afni--23.3.02.simg ok INFO: file images/neurodesk/neurodesk-afni--24.1.02.simg OK: https://d15yxasja65rk8.cloudfront.net/afni_24.1.02_20240409.simg INFO: adding https://d15yxasja65rk8.cloudfront.net/afni_24.1.02_20240409.simg to images/neurodesk/neurodesk-afni--24.1.02.simg ^[[0Jaddurl https://d15yxasja65rk8.cloudfront.net/afni_24.1.02_20240409.simg ok ^[[1G INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--24.1.02.simg ok rmurl images/neurodesk/neurodesk-afni--24.1.02.simg ok rmurl images/neurodesk/neurodesk-afni--24.1.02.simg ok INFO: file images/neurodesk/neurodesk-aidamri--1.1.simg OK: https://d15yxasja65rk8.cloudfront.net/aidamri_1.1_20210708.simg INFO: adding https://d15yxasja65rk8.cloudfront.net/aidamri_1.1_20210708.simg to images/neurodesk/neurodesk-aidamri--1.1.simg ^[[0Jaddurl https://d15yxasja65rk8.cloudfront.net/aidamri_1.1_20210708.simg ok ^[[1G INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-aidamri--1.1.simg ok rmurl images/neurodesk/neurodesk-aidamri--1.1.simg ok rmurl images/neurodesk/neurodesk-aidamri--1.1.simg ok INFO: file images/neurodesk/neurodesk-ants--2.3.1.simg OK: https://d15yxasja65rk8.cloudfront.net/ants_2.3.1_20211204.simg INFO: adding https://d15yxasja65rk8.cloudfront.net/ants_2.3.1_20211204.simg to images/neurodesk/neurodesk-ants--2.3.1.simg ^[[0Jaddurl https://d15yxasja65rk8.cloudfront.net/ants_2.3.1_20211204.simg ok ^[[1G INFO: removing 3 oracle urls rmurl images/neurodesk/neuERROR: https://d15yxasja65rk8.cloudfront.net/ants_2.3.4_20211212.simg - could not verify presence: 403 . Will not be added OK: https://d15yxasja65rk8.cloudfront.net/ants_2.3.5_20211212.simg OK: https://d15yxasja65rk8.cloudfront.net/ashs_2.0.0_20210105.simg OK: https://d15yxasja65rk8.cloudfront.net/aslprep_0.2.7_20210323.simg OK: https://d15yxasja65rk8.cloudfront.net/aslprep_0.4.0_20230728.simg OK: https://d15yxasja65rk8.cloudfront.net/aslprep_0.5.0_20231116.simg ERROR: https://d15yxasja65rk8.cloudfront.net/aslprep_0.6.0_20240129.simg - could not verify presence: 403 . Will not be added OK: https://d15yxasja65rk8.cloudfront.net/bart_0.7.00_20210302.simg ERROR: https://d15yxasja65rk8.cloudfront.net/bidsappaa_0.2.0_20230612.simg - could not verify presence: 403 . Will not be added OK: https://d15yxasja65rk8.cloudfront.net/bidsappbaracus_1.1.4_20230612.simg ERROR: https://d15yxasja65rk8.cloudfront.net/bidsappbrainsuite_21a_20230615.simg - could not verify presence: 403 . Will not be added OK: https://d15yxasja65rk8.cloudfront.net/bidsapphcppipelines_4.3.0_20230524.simg OK: https://d15yxasja65rk8.cloudfront.net/bidsappmrtrix3connectome_0.5.3_20230615.simg OK: https://d15yxasja65rk8.cloudfront.net/bidsapppymvpa_2.0.2_20230629.simg OK: https://d15yxasja65rk8.cloudfront.net/bidsappspm_0.0.20_20230629.simg OK: https://d15yxasja65rk8.cloudfront.net/bidscoin_3.7.0_20220329.simg OK: https://d15yxasja65rk8.cloudfront.net/bidscoin_4.2.0_20231017.simg OK: https://d15yxasja65rk8.cloudfront.net/bidscoin_4.2.1_20231030.simg ERROR: https://d15yxasja65rk8.cloudfront.net/bidscoin_4.3.0_20240220.simg - could not verify presence: 403 . Will not be added OK: https://d15yxasja65rk8.cloudfront.net/bidscoin_4.3.2_20240329.simg ERROR: https://d15yxasja65rk8.cloudfront.net/bidstools_1.0.0_20201208.simg - could not verify presence: 403 . Will not be added OK: https://d15yxasja65rk8.cloudfront.net/bidstools_1.0.1_20230905.simg OK: https://d15yxasja65rk8.cloudfront.net/bidstools_1.0.2_20231017.simg OK: https://d15yxasja65rk8.cloudfront.net/bidstools_1.0.3_20231030.simg OK: https://d15yxasja65rk8.cloudfront.net/bidstools_1.0.4_20240221.simg OK: https://d15yxasja65rk8.cloudfront.net/brainstorm_3.211130_20211207.simg ERROR: https://d15yxasja65rk8.cloudfront.net/brkraw_0.3.11_20240223.simg - could not verify presence: 403 . Will not be added OK: https://d15yxasja65rk8.cloudfront.net/cat12_r1933_20220128.simg OK: https://d15yxasja65rk8.cloudfront.net/cat12_r2166_20230601.simg OK: https://d15yxasja65rk8.cloudfront.net/clearswi_1.0.0_20211018.simg ERROR: https://d15yxasja65rk8.cloudfront.net/code_220114_20220121.simg - could not verify presence: 403 . Will not be added OK: https://d15yxasja65rk8.cloudfront.net/code_230315_20230315.simg OK: https://d15yxasja65rk8.cloudfront.net/conn_20b_20210109.simg OK: https://d15yxasja65rk8.cloudfront.net/conn_22a_20231115.simg error: git-annex died of signal 9 {"Command": "scripts/replace_neurodesk_urls", "System": {"uid": "yoh", "memory_total": 135060111360, "cpu_total": 16}, "ENV": [{}], "GPU": []} [INFO ] == Command exit (modification check follows) ===== run(ok): /home/yoh/proj/repronim/containers (dataset) [duct scripts/replace_neurodesk_urls] add(ok): .duct/logs/2024-05-06T13-29-22-390727_stderr (file) add(ok): .duct/logs/2024-05-06T13-29-22-390727_stdout (file) add(ok): .duct/logs/2024-05-06T13-29-22-390727_info.json (file) save(ok): . (dataset) datalad run -m "Update neurodesk image urls" duct 26.46s user 16.71s system 0% cpu 1:28:35.37 total ```

note: I had to kill underlying process since it stalled... not sure yet either due to duct "clogging" outputs or not, but FTR

here is the stack of processes ```shell yoh 390634 0.0 0.0 283344 52372 pts/7 Sl+ 13:29 0:01 /usr/bin/python3 /usr/bin/datalad run -m Update neurodesk image urls duct scripts/replace_neurodesk_urls yoh 390725 0.0 0.0 2580 1536 pts/7 S+ 13:29 0:00 /bin/sh -c duct scripts/replace_neurodesk_urls yoh 390727 0.0 0.0 242704 13952 pts/7 Sl+ 13:29 0:00 /home/yoh/venvs/dev3/bin/python3 /home/yoh/venvs/dev3/bin/duct scripts/replace_neurodesk_urls yoh 390730 0.0 0.0 7084 3328 ? Ss 13:29 0:00 /bin/bash scripts/replace_neurodesk_urls yoh 398627 0.0 0.0 9896 3712 ? S 13:29 0:00 git -c annex.alwayscommit=false annex addurl --file=images/neurodesk/neurodesk-conn--22a.simg https://d15yxasja65rk8.cloudfront.net/conn_22a_20231115.simg yoh 398628 0.0 0.0 1075011040 66704 ? Sl 13:29 0:00 /home/yoh/git-annexes/10.20240430+git26-g5f61667f27/usr/lib/git-annex.linux/exe/git-annex --library-path /home/yoh/git-annexes/10.20240430+git26-g5f61667f27/usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /home/yoh/git-annexes/10.20240430+git26-g5f61667f27/usr/lib/git-annex.linux/shimmed/git-annex/git-annex addurl --file=images/neurodesk/neurodesk-conn--22a.simg https://d15yxasja65rk8.cloudfront.net/conn_22a_20231115.simg yoh 398658 0.0 0.0 11208 5248 ? S 13:29 0:00 /home/yoh/git-annexes/10.20240430+git26-g5f61667f27/usr/lib/git-annex.linux/exe/git --library-path /home/yoh/git-annexes/10.20240430+git26-g5f61667f27/usr/lib/git-annex.linux//lib/x86_64-linux-gnu: /home/yoh/git-annexes/10.20240430+git26-g5f61667f27/usr/lib/git-annex.linux/shimmed/git/git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch ```

after I killed that addurl we got the process terminated

but stdout was very incomplete if we compare to above ```shell $> cat .duct/logs/2024-05-06T13-29-22-390727_stdout INFO: file images/neurodesk/neurodesk-afni--21.2.00.simg INFO: adding https://d15yxasja65rk8.cloudfront.net/afni_21.2.00_20210714.simg to images/neurodesk/neurodesk-afni--21.2.00.simg ^[[0Jaddurl https://d15yxasja65rk8.cloudfront.net/afni_21.2.00_20210714.simg ok ^[[1G INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--21.2.00.simg ok rmurl images/neurodesk/neurodesk-afni--21.2.00.simg ok rmurl images/neurodesk/neurodesk-afni--21.2.00.simg ok INFO: file images/neurodesk/neurodesk-afni--22.1.14.simg INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--22.1.14.simg ok rmurl images/neurodesk/neurodesk-afni--22.1.14.simg ok rmurl images/neurodesk/neurodesk-afni--22.1.14.simg ok INFO: file images/neurodesk/neurodesk-afni--22.3.06.simg INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--22.3.06.simg ok rmurl images/neurodesk/neurodesk-afni--22.3.06.simg ok rmurl images/neurodesk/neurodesk-afni--22.3.06.simg ok INFO: file images/neurodesk/neurodesk-afni--22.3.07.simg INFO: adding https://d15yxasja65rk8.cloudfront.net/afni_22.3.07_20221206.simg to images/neurodesk/neurodesk-afni--22.3.07.simg ^[[0Jaddurl https://d15yxasja65rk8.cloudfront.net/afni_22.3.07_20221206.simg ok ^[[1G INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--22.3.07.simg ok rmurl images/neurodesk/neurodesk-afni--22.3.07.simg ok rmurl images/neurodesk/neurodesk-afni--22.3.07.simg ok INFO: file images/neurodesk/neurodesk-afni--23.0.00.simg INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--23.0.00.simg ok rmurl images/neurodesk/neurodesk-afni--23.0.00.simg ok rmurl images/neurodesk/neurodesk-afni--23.0.00.simg ok INFO: file images/neurodesk/neurodesk-afni--23.0.04.simg INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--23.0.04.simg ok rmurl images/neurodesk/neurodesk-afni--23.0.04.simg ok rmurl images/neurodesk/neurodesk-afni--23.0.04.simg ok INFO: file images/neurodesk/neurodesk-afni--23.0.07.simg INFO: adding https://d15yxasja65rk8.cloudfront.net/afni_23.0.07_20230302.simg to images/neurodesk/neurodesk-afni--23.0.07.simg ^[[0Jaddurl https://d15yxasja65rk8.cloudfront.net/afni_23.0.07_20230302.simg ok ^[[1G INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--23.0.07.simg ok rmurl images/neurodesk/neurodesk-afni--23.0.07.simg ok rmurl images/neurodesk/neurodesk-afni--23.0.07.simg ok INFO: file images/neurodesk/neurodesk-afni--23.3.02.simg INFO: adding https://d15yxasja65rk8.cloudfront.net/afni_23.3.02_20231024.simg to images/neurodesk/neurodesk-afni--23.3.02.simg ^[[0Jaddurl https://d15yxasja65rk8.cloudfront.net/afni_23.3.02_20231024.simg ok ^[[1G INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--23.3.02.simg ok rmurl images/neurodesk/neurodesk-afni--23.3.02.simg ok rmurl images/neurodesk/neurodesk-afni--23.3.02.simg ok INFO: file images/neurodesk/neurodesk-afni--24.1.02.simg INFO: adding https://d15yxasja65rk8.cloudfront.net/afni_24.1.02_20240409.simg to images/neurodesk/neurodesk-afni--24.1.02.simg ^[[0Jaddurl https://d15yxasja65rk8.cloudfront.net/afni_24.1.02_20240409.simg ok ^[[1G INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-afni--24.1.02.simg ok rmurl images/neurodesk/neurodesk-afni--24.1.02.simg ok rmurl images/neurodesk/neurodesk-afni--24.1.02.simg ok INFO: file images/neurodesk/neurodesk-aidamri--1.1.simg INFO: adding https://d15yxasja65rk8.cloudfront.net/aidamri_1.1_20210708.simg to images/neurodesk/neurodesk-aidamri--1.1.simg ^[[0Jaddurl https://d15yxasja65rk8.cloudfront.net/aidamri_1.1_20210708.simg ok ^[[1G INFO: removing 3 oracle urls rmurl images/neurodesk/neurodesk-aidamri--1.1.simg ok rmurl images/neurodesk/neurodesk-aidamri--1.1.simg ok rmurl images/neurodesk/neurodesk-aidamri--1.1.simg ok INFO: file images/neurodesk/neurodesk-ants--2.3.1.simg INFO: adding https://d15yxasja65rk8.cloudfront.net/ants_2.3.1_20211204.simg to images/neurodesk/neurodesk-ants--2.3.1.simg ^[[0Jaddurl https://d15yxasja65rk8.cloudfront.net/ants_2.3.1_20211204.simg ok ^[[1G INFO: removing 3 oracle urls rmurl images/neurodesk/neu% ```

so probably buffered and buffer was not flushed etc.