search

theiagen / public_health_bioinformatics

Bioinformatics workflows for genomic characterization, submission preparation, and genomic epidemiology of pathogens of public health concern.
GNU General Public License v3.0
37 stars 17 forks source link

[PHB Release 1.3.0] update version and docker tags (nexclade sc2, pangolin, tbp-parser 1.1.7) #296

Closed cimendes closed 9 months ago

cimendes commented 9 months ago

Closes

:hammer_and_wrench: Changes Being Made

Impacted Workflows/Tasks

:brain: Context and Rationale

:clipboard: Workflow/Task Steps

Inputs

Outputs

Impacted Outputs

:test_tube: Testing

Locally

Terra

Scenarios for Reviewer to Test

:microscope: Quality checks

Pull Request (PR) checklist:

kapsakcj commented 9 months ago

Running cromwell tests locally on my devVM (8cpus 32GB RAM, ample storage available), using an identical conda/mamba environment.

These commands were run inside the public_health_bioinformatics root directory, so all WDLs and other files are available:

Environment setup: ```bash # check that mamba is installed $ mamba --version mamba 1.4.2 conda 23.3.1 # already have docker installed locally # install CI environment that includes cromwell (and pytest and other stuff) $ mamba env create --name pytest-cromwell-CI-env --f ile tests/config/environment.yml Retrieving notices: ...working... done bioconda/noarch 5.1MB @ 3.6MB/s 1.6s pkgs/main/linux-64 6.5MB @ 3.8MB/s 1.9s bioconda/linux-64 5.3MB @ 2.7MB/s 2.1s pkgs/r/noarch No change pkgs/r/linux-64 No change nodefaults/linux-64 No change pkgs/main/noarch 855.2kB @ 358.0kB/s 1.0s nodefaults/noarch No change conda-forge/noarch 15.3MB @ 5.9MB/s 2.9s conda-forge/linux-64 37.2MB @ 6.9MB/s 6.3s Looking for: ["python[version='>=3.7']", 'cromwell', 'miniwdl=1.5.2', 'pytest', 'pytest-workflow', "importlib-metadata[version='<=4.13.0']"] Transaction Prefix: /home/curtis_kapsak/miniforge3/envs/pytest-cromwell-CI-env Updating specs: - python[version='>=3.7'] - cromwell - miniwdl=1.5.2 - pytest - pytest-workflow - importlib-metadata[version='<=4.13.0'] Package Version Build Channel Size ────────────────────────────────────────────────────────────────────────────────────────────────── Install: ────────────────────────────────────────────────────────────────────────────────────────────────── + _libgcc_mutex 0.1 conda_forge conda-forge/linux-64 Cached + _openmp_mutex 4.5 2_gnu conda-forge/linux-64 Cached + alsa-lib 1.2.10 hd590300_0 conda-forge/linux-64 Cached + argcomplete 3.2.1 pyhd8ed1ab_0 conda-forge/noarch 40kB + atomicwrites 1.4.1 pyhd8ed1ab_0 conda-forge/noarch Cached + attrs 23.2.0 pyh71513ae_0 conda-forge/noarch 55kB + bcrypt 4.1.2 py312h4b3b743_0 conda-forge/linux-64 579kB + brotli-python 1.1.0 py312h30efb56_1 conda-forge/linux-64 Cached + bullet-python 2.2.0 py312h7900ff3_7 conda-forge/linux-64 Cached + bzip2 1.0.8 hd590300_5 conda-forge/linux-64 Cached + ca-certificates 2023.11.17 hbcca054_0 conda-forge/linux-64 Cached + cairo 1.18.0 h3faef2a_0 conda-forge/linux-64 Cached + certifi 2023.11.17 pyhd8ed1ab_0 conda-forge/noarch Cached + cffi 1.16.0 py312hf06ca03_0 conda-forge/linux-64 Cached + charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge/noarch Cached + colorama 0.4.6 pyhd8ed1ab_0 conda-forge/noarch Cached + coloredlogs 15.0.1 pyhd8ed1ab_3 conda-forge/noarch Cached + cromwell 83 ha770c72_0 conda-forge/linux-64 Cached + cryptography 41.0.7 py312h4742d6a_1 conda-forge/linux-64 2MB + docker-py 7.0.0 pyhd8ed1ab_0 conda-forge/noarch 103kB + exceptiongroup 1.2.0 pyhd8ed1ab_0 conda-forge/noarch 20kB + expat 2.5.0 hcb278e6_1 conda-forge/linux-64 Cached + findutils 4.6.0 h166bdaf_1001 conda-forge/linux-64 Cached + font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge/noarch Cached + font-ttf-inconsolata 3.000 h77eed37_0 conda-forge/noarch Cached + font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge/noarch Cached + font-ttf-ubuntu 0.83 h77eed37_1 conda-forge/noarch 2MB + fontconfig 2.14.2 h14ed4e7_0 conda-forge/linux-64 Cached + fonts-conda-ecosystem 1 0 conda-forge/noarch Cached + fonts-conda-forge 1 0 conda-forge/noarch Cached + freetype 2.12.1 h267a509_2 conda-forge/linux-64 Cached + gettext 0.21.1 h27087fc_0 conda-forge/linux-64 Cached + giflib 5.2.1 h0b41bf4_3 conda-forge/linux-64 Cached + graphite2 1.3.13 h58526e2_1001 conda-forge/linux-64 Cached + harfbuzz 8.3.0 h3d44ed6_0 conda-forge/linux-64 Cached + humanfriendly 10.0 pyhd8ed1ab_6 conda-forge/noarch Cached + icu 73.2 h59595ed_0 conda-forge/linux-64 Cached + idna 3.6 pyhd8ed1ab_0 conda-forge/noarch 50kB + importlib-metadata 4.13.0 pyha770c72_0 conda-forge/noarch Cached + importlib_metadata 4.13.0 hd8ed1ab_0 conda-forge/noarch Cached + importlib_resources 6.1.1 pyhd8ed1ab_0 conda-forge/noarch Cached + iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge/noarch Cached + js2py 0.74 pyhd8ed1ab_0 conda-forge/noarch Cached + jsonschema 4.20.0 pyhd8ed1ab_0 conda-forge/noarch Cached + jsonschema-specifications 2023.12.1 pyhd8ed1ab_0 conda-forge/noarch 16kB + keyutils 1.6.1 h166bdaf_0 conda-forge/linux-64 Cached + krb5 1.21.2 h659d440_0 conda-forge/linux-64 Cached + lark 1.1.2 pyhd8ed1ab_0 conda-forge/noarch Cached + lcms2 2.16 hb7c19ff_0 conda-forge/linux-64 245kB + ld_impl_linux-64 2.40 h41732ed_0 conda-forge/linux-64 Cached + lerc 4.0.0 h27087fc_0 conda-forge/linux-64 Cached + libcups 2.3.3 h4637d8d_4 conda-forge/linux-64 Cached + libdeflate 1.19 hd590300_0 conda-forge/linux-64 Cached + libedit 3.1.20191231 he28a2e2_2 conda-forge/linux-64 Cached + libexpat 2.5.0 hcb278e6_1 conda-forge/linux-64 Cached + libffi 3.4.2 h7f98852_5 conda-forge/linux-64 Cached + libgcc-ng 13.2.0 h807b86a_3 conda-forge/linux-64 Cached + libglib 2.78.3 h783c2da_0 conda-forge/linux-64 3MB + libgomp 13.2.0 h807b86a_3 conda-forge/linux-64 Cached + libiconv 1.17 hd590300_2 conda-forge/linux-64 706kB + libjpeg-turbo 3.0.0 hd590300_1 conda-forge/linux-64 Cached + libnsl 2.0.1 hd590300_0 conda-forge/linux-64 Cached + libpng 1.6.39 h753d276_0 conda-forge/linux-64 Cached + libsodium 1.0.18 h36c2ea0_1 conda-forge/linux-64 Cached + libsqlite 3.44.2 h2797004_0 conda-forge/linux-64 846kB + libstdcxx-ng 13.2.0 h7e041cc_3 conda-forge/linux-64 Cached + libtiff 4.6.0 ha9c0a0a_2 conda-forge/linux-64 Cached + libuuid 2.38.1 h0b41bf4_0 conda-forge/linux-64 Cached + libwebp-base 1.3.2 hd590300_0 conda-forge/linux-64 Cached + libxcb 1.15 h0b41bf4_0 conda-forge/linux-64 Cached + libxcrypt 4.4.36 hd590300_1 conda-forge/linux-64 100kB + libzlib 1.2.13 hd590300_5 conda-forge/linux-64 Cached + miniwdl 1.5.2 pyhd8ed1ab_0 conda-forge/noarch Cached + ncurses 6.4 h59595ed_2 conda-forge/linux-64 Cached + openjdk 11.0.21 h4260e57_0 conda-forge/linux-64 Cached + openssl 3.2.0 hd590300_1 conda-forge/linux-64 3MB + packaging 23.2 pyhd8ed1ab_0 conda-forge/noarch Cached + paramiko 3.4.0 pyhd8ed1ab_0 conda-forge/noarch 160kB + pcre2 10.42 hcad00b1_0 conda-forge/linux-64 Cached + pip 23.3.2 pyhd8ed1ab_0 conda-forge/noarch 1MB + pixman 0.43.0 h59595ed_0 conda-forge/linux-64 387kB + pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge/noarch Cached + pluggy 1.3.0 pyhd8ed1ab_0 conda-forge/noarch Cached + psutil 5.9.7 py312h98912ed_0 conda-forge/linux-64 485kB + pthread-stubs 0.4 h36c2ea0_1001 conda-forge/linux-64 Cached + pycparser 2.21 pyhd8ed1ab_0 conda-forge/noarch Cached + pygtail 0.14.0 pyhd8ed1ab_0 conda-forge/noarch Cached + pyjsparser 2.7.1 pyh8c360ce_0 conda-forge/noarch Cached + pynacl 1.5.0 py312h98912ed_3 conda-forge/linux-64 Cached + pysocks 1.7.1 pyha2e5f31_6 conda-forge/noarch Cached + pytest 7.4.4 pyhd8ed1ab_0 conda-forge/noarch 245kB + pytest-workflow 2.0.1 pyhd8ed1ab_0 conda-forge/noarch Cached + python 3.12.1 hab00c5b_1_cpython conda-forge/linux-64 32MB + python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge/noarch Cached + python_abi 3.12 4_cp312 conda-forge/linux-64 Cached + pywin32-on-windows 0.1.0 pyh1179c8e_3 conda-forge/noarch Cached + pyyaml 6.0.1 py312h98912ed_1 conda-forge/linux-64 Cached + readline 8.2 h8228510_1 conda-forge/linux-64 Cached + referencing 0.32.1 pyhd8ed1ab_0 conda-forge/noarch 39kB + regex 2023.12.25 py312h98912ed_0 conda-forge/linux-64 399kB + requests 2.31.0 pyhd8ed1ab_0 conda-forge/noarch Cached + rpds-py 0.16.2 py312h4b3b743_0 conda-forge/linux-64 998kB + setuptools 69.0.3 pyhd8ed1ab_0 conda-forge/noarch 471kB + six 1.16.0 pyh6c4a22f_0 conda-forge/noarch Cached + tk 8.6.13 noxft_h4845f30_101 conda-forge/linux-64 Cached + tomli 2.0.1 pyhd8ed1ab_0 conda-forge/noarch Cached + tzdata 2023d h0c530f3_0 conda-forge/noarch 120kB + tzlocal 5.2 py312h7900ff3_0 conda-forge/linux-64 Cached + urllib3 2.1.0 pyhd8ed1ab_0 conda-forge/noarch Cached + websocket-client 1.7.0 pyhd8ed1ab_0 conda-forge/noarch 47kB + wheel 0.42.0 pyhd8ed1ab_0 conda-forge/noarch 58kB + xdg 6.0.0 pyhd8ed1ab_0 conda-forge/noarch Cached + xorg-fixesproto 5.0 h7f98852_1002 conda-forge/linux-64 Cached + xorg-inputproto 2.3.2 h7f98852_1002 conda-forge/linux-64 Cached + xorg-kbproto 1.0.7 h7f98852_1002 conda-forge/linux-64 Cached + xorg-libice 1.1.1 hd590300_0 conda-forge/linux-64 Cached + xorg-libsm 1.2.4 h7391055_0 conda-forge/linux-64 Cached + xorg-libx11 1.8.7 h8ee46fc_0 conda-forge/linux-64 Cached + xorg-libxau 1.0.11 hd590300_0 conda-forge/linux-64 Cached + xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge/linux-64 Cached + xorg-libxext 1.3.4 h0b41bf4_2 conda-forge/linux-64 Cached + xorg-libxfixes 5.0.3 h7f98852_1004 conda-forge/linux-64 Cached + xorg-libxi 1.7.10 h7f98852_0 conda-forge/linux-64 Cached + xorg-libxrender 0.9.11 hd590300_0 conda-forge/linux-64 Cached + xorg-libxt 1.3.0 hd590300_1 conda-forge/linux-64 Cached + xorg-libxtst 1.2.3 h7f98852_1002 conda-forge/linux-64 Cached + xorg-recordproto 1.14.2 h7f98852_1002 conda-forge/linux-64 Cached + xorg-renderproto 0.11.1 h7f98852_1002 conda-forge/linux-64 Cached + xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge/linux-64 Cached + xorg-xproto 7.0.31 h7f98852_1007 conda-forge/linux-64 Cached + xz 5.2.6 h166bdaf_0 conda-forge/linux-64 Cached + yaml 0.2.5 h7f98852_2 conda-forge/linux-64 Cached + zipp 3.17.0 pyhd8ed1ab_0 conda-forge/noarch Cached + zlib 1.2.13 hd590300_5 conda-forge/linux-64 Cached + zstd 1.5.5 hfc55251_0 conda-forge/linux-64 Cached Summary: Install: 135 packages Total download: 49MB ────────────────────────────────────────────────────────────────────────────────────────────────── libxcrypt 100.4kB @ 762.7kB/s 0.1s pixman 387.3kB @ 2.7MB/s 0.1s wheel 57.6kB @ 332.1kB/s 0.0s libsqlite 845.8kB @ 4.6MB/s 0.2s libiconv 705.8kB @ 3.8MB/s 0.2s openssl 2.9MB @ 13.5MB/s 0.2s idna 50.1kB @ 237.2kB/s 0.1s attrs 54.6kB @ 242.5kB/s 0.1s regex 398.6kB @ 1.7MB/s 0.1s jsonschema-specifications 16.4kB @ 62.5kB/s 0.1s paramiko 160.0kB @ 604.3kB/s 0.1s bcrypt 579.3kB @ 2.0MB/s 0.1s pip 1.4MB @ 4.5MB/s 0.1s exceptiongroup 20.5kB @ 66.3kB/s 0.0s font-ttf-ubuntu 1.6MB @ 5.0MB/s 0.1s docker-py 103.5kB @ 300.7kB/s 0.0s rpds-py 998.1kB @ 2.9MB/s 0.1s tzdata 119.6kB @ 334.8kB/s 0.1s psutil 485.3kB @ 1.3MB/s 0.1s websocket-client 46.6kB @ 107.9kB/s 0.1s setuptools 470.5kB @ 1.1MB/s 0.1s libglib 2.7MB @ 5.9MB/s 0.1s cryptography 2.0MB @ 4.5MB/s 0.1s lcms2 245.2kB @ 507.1kB/s 0.0s referencing 39.0kB @ 79.9kB/s 0.0s argcomplete 40.1kB @ 78.4kB/s 0.1s pytest 244.6kB @ 477.3kB/s 0.1s python 32.3MB @ 49.4MB/s 0.3s Downloading and Extracting Packages Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate pytest-cromwell-CI-env # # To deactivate an active environment, use # # $ conda deactivate # activate environment $ mamba activate pytest-cromwell-CI-env ```
TheiaCoV_Fasta test via `cromwell`: ```bash $ cromwell run -i ./tests/inputs/theiacov/wf_theiacov_fasta.json - m metadata.json ./workflows/theiacov/wf_theiacov_fasta.wdl [2024-01-10 13:55:05,83] [info] Running with database db.url = jdbc:hsqldb:mem:10753a5b-27ec-4041-b012-bbc749a5b78e;shutdown=false;hsqldb.tx=mvcc Jan 10, 2024 1:55:06 PM liquibase.servicelocator INFO: Cannot load service: liquibase.license.LicenseService: liquibase.license.pro.DaticalTrueLicenseService Unable to get public no-arg constructor Jan 10, 2024 1:55:06 PM liquibase.database INFO: Set default schema name to PUBLIC Jan 10, 2024 1:55:06 PM liquibase.lockservice INFO: Successfully acquired change log lock Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Creating database history table with name: PUBLIC.DATABASECHANGELOG Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Reading from PUBLIC.DATABASECHANGELOG Running Changeset: changesets/db_schema.xml::db_schema_other_table_alldb::scottfrazer Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Table WORKFLOW_EXECUTION created Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Table EXECUTION created Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Table JES_JOB created Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Table LOCAL_JOB created Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/db_schema.xml::db_schema_other_table_alldb::scottfrazer ran successfully in 20ms Running Changeset: changesets/db_schema.xml::db_schema_symbol_table_otherdb::scottfrazer Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Table SYMBOL created Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/db_schema.xml::db_schema_symbol_table_otherdb::scottfrazer ran successfully in 5ms Running Changeset: changesets/db_schema.xml::db_schema_constraints::scottfrazer Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Index STATUS_IDX created Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Foreign key constraint added to SYMBOL (WORKFLOW_EXECUTION_ID) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION (WORKFLOW_EXECUTION_ID) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Foreign key constraint added to JES_JOB (EXECUTION_ID) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Foreign key constraint added to LOCAL_JOB (EXECUTION_ID) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/db_schema.xml::db_schema_constraints::scottfrazer ran successfully in 22ms Running Changeset: changesets/symbol_iteration_null.xml::symbol_iteration_null::kshakir Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Null constraint dropped from SYMBOL.ITERATION Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/symbol_iteration_null.xml::symbol_iteration_null::kshakir ran successfully in 3ms Running Changeset: changesets/wdl_and_inputs.xml::WORKFLOW_EXECUTION_AUX::mcovarr Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Table WORKFLOW_EXECUTION_AUX created Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/wdl_and_inputs.xml::WORKFLOW_EXECUTION_AUX::mcovarr ran successfully in 4ms Running Changeset: changesets/wdl_and_inputs.xml::WE_AUX_CONSTRAINTS::mcovarr Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Foreign key constraint added to WORKFLOW_EXECUTION_AUX (WORKFLOW_EXECUTION_ID) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_EXECUTION_AUX(WORKFLOW_EXECUTION_ID) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/wdl_and_inputs.xml::WE_AUX_CONSTRAINTS::mcovarr ran successfully in 7ms Running Changeset: changesets/drop_workflow_uri_and_local_command.xml::drop_workflow_uri_and_local_command::kshakir Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Column WORKFLOW_EXECUTION.WDL_URI dropped Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Column LOCAL_JOB.COMMAND dropped Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/drop_workflow_uri_and_local_command.xml::drop_workflow_uri_and_local_command::kshakir ran successfully in 6ms Running Changeset: changesets/local_job_allow_null.xml::local_job_allow_null::kshakir Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Null constraint dropped from LOCAL_JOB.PID Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Null constraint dropped from LOCAL_JOB.RC Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/local_job_allow_null.xml::local_job_allow_null::kshakir ran successfully in 5ms Running Changeset: changesets/symbol_iteration_not_null.xml::symbol_iteration_not_null::kshakir Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: NOT NULL constraint has been added to SYMBOL.ITERATION Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/symbol_iteration_not_null.xml::symbol_iteration_not_null::kshakir ran successfully in 5ms Running Changeset: changesets/add_unique_constraints.xml::add_unique_constraints::kshakir Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_EXECUTION(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Unique constraint added to EXECUTION(WORKFLOW_EXECUTION_ID, CALL_FQN) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Unique constraint added to SYMBOL(WORKFLOW_EXECUTION_ID, SCOPE, NAME, ITERATION, IO) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Unique constraint added to LOCAL_JOB(EXECUTION_ID) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Unique constraint added to JES_JOB(EXECUTION_ID) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/add_unique_constraints.xml::add_unique_constraints::kshakir ran successfully in 16ms Running Changeset: changesets/lengthen_wdl_value.xml::LENGTHEN_WDL_VALUE::sfrazer Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: SYMBOL.WDL_VALUE datatype was changed to LONGVARCHAR Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/lengthen_wdl_value.xml::LENGTHEN_WDL_VALUE::sfrazer ran successfully in 4ms Running Changeset: changesets/add_index_in_execution.xml::add-index-in-execution::tjeandet Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Columns INDEX(INT) added to EXECUTION Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: NOT NULL constraint has been added to EXECUTION.INDEX Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/add_index_in_execution.xml::add-index-in-execution::tjeandet ran successfully in 6ms Running Changeset: changesets/rename_iteration_to_index.xml::rename-iteration-to-index::tjeandet Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Column SYMBOL.ITERATION renamed to INDEX Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/rename_iteration_to_index.xml::rename-iteration-to-index::tjeandet ran successfully in 2ms Running Changeset: changesets/sge.xml::sge::scottfrazer Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Table SGE_JOB created Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/sge.xml::sge::scottfrazer ran successfully in 3ms Running Changeset: changesets/sge.xml::db_schema_constraints::scottfrazer Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Foreign key constraint added to SGE_JOB (EXECUTION_ID) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/sge.xml::db_schema_constraints::scottfrazer ran successfully in 3ms Running Changeset: changesets/change_execution_unique_constraint.xml::change_execution_unique_constraint::tjeandet Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Foreign key FK_EXECUTION_WORKFLOW_EXECUTION_ID dropped Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Unique constraint UK_EX_WORKFLOW_EXECUTION_ID dropped from EXECUTION Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Unique constraint added to EXECUTION(WORKFLOW_EXECUTION_ID, CALL_FQN, INDEX) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION (WORKFLOW_EXECUTION_ID) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Column EXECUTION.INDEX renamed to IDX Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/change_execution_unique_constraint.xml::change_execution_unique_constraint::tjeandet ran successfully in 20ms Running Changeset: changesets/rc.xml::rc::mcovarr Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Column LOCAL_JOB.RC dropped Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Columns RC(INT) added to EXECUTION Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/rc.xml::rc::mcovarr ran successfully in 8ms Running Changeset: changesets/workflow_options.xml::workflow-options::sfrazer Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Columns WORKFLOW_OPTIONS(LONGVARCHAR) added to WORKFLOW_EXECUTION_AUX Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/workflow_options.xml::workflow-options::sfrazer ran successfully in 4ms Running Changeset: changesets/jes_id_update.xml::jes_id_update::chrisl Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: JES_JOB.JES_ID datatype was changed to VARCHAR(100) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: JES_JOB.JES_STATUS datatype was changed to VARCHAR(100) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/jes_id_update.xml::jes_id_update::chrisl ran successfully in 4ms Running Changeset: changesets/optional_sge_ids.xml::optional_sge_ids::chrisl Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: SGE_JOB.SGE_JOB_NUMBER datatype was changed to INT Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/optional_sge_ids.xml::optional_sge_ids::chrisl ran successfully in 2ms Running Changeset: changesets/add_start_end_time_in_execution.xml::add-start-end-time-in-execution::jgentry Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Columns START_DT(datetime) added to EXECUTION Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Columns END_DT(datetime) added to EXECUTION Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/add_start_end_time_in_execution.xml::add-start-end-time-in-execution::jgentry ran successfully in 4ms Running Changeset: changesets/rename_jes_id.xml::rename-jes-id-to-jes-run-id::sfrazer Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Column JES_JOB.JES_ID renamed to JES_RUN_ID Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/rename_jes_id.xml::rename-jes-id-to-jes-run-id::sfrazer ran successfully in 2ms Running Changeset: changesets/add_workflow_name.xml::workflow-name::mcovarr Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Columns WORKFLOW_NAME(VARCHAR(500)) added to WORKFLOW_EXECUTION Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/add_workflow_name.xml::workflow-name::mcovarr ran successfully in 4ms Running Changeset: changesets/top_level_output.xml::top_level_output::chrisl Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Columns REPORTABLE_RESULT(TINYINT) added to SYMBOL Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/top_level_output.xml::top_level_output::chrisl ran successfully in 3ms Running Changeset: changesets/workflow_execution_aux_not_null.xml::workflow_execution_aux_not_null::tjeandet Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_EXECUTION_AUX.WORKFLOW_EXECUTION_ID Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_EXECUTION_AUX.WDL_SOURCE Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_EXECUTION_AUX.JSON_INPUTS Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_EXECUTION_AUX.WORKFLOW_OPTIONS Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/workflow_execution_aux_not_null.xml::workflow_execution_aux_not_null::tjeandet ran successfully in 9ms Running Changeset: changesets/call_result_caching.xml::call_result_caching::chrisl Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Columns ALLOWS_RESULT_REUSE(BOOLEAN),DOCKER_IMAGE_HASH(VARCHAR(100)),RESULTS_CLONED_FROM(INT),EXECUTION_HASH(VARCHAR(100)) added to EXECUTION Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Index HASH_INDEX created Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Columns HASH(VARCHAR(100)) added to SYMBOL Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION (RESULTS_CLONED_FROM) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/call_result_caching.xml::call_result_caching::chrisl ran successfully in 13ms Running Changeset: changesets/events_table.xml::execution_event_table::chrisl Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Table EXECUTION_EVENT created Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION_EVENT (EXECUTION_ID) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Unique constraint added to EXECUTION_EVENT(EXECUTION_ID, DESCRIPTION) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/events_table.xml::execution_event_table::chrisl ran successfully in 10ms Running Changeset: changesets/sync_not_null_constraints.xml::job_not_null_columns::kshakir Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Null constraint dropped from JES_JOB.JES_RUN_ID Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Null constraint dropped from JES_JOB.JES_STATUS Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Null constraint dropped from SGE_JOB.SGE_JOB_NUMBER Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/sync_not_null_constraints.xml::job_not_null_columns::kshakir ran successfully in 6ms Running Changeset: changesets/sync_not_null_constraints.xml::events_end_dt_not_null::kshakir Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: NOT NULL constraint has been added to EXECUTION_EVENT.END_DT Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/sync_not_null_constraints.xml::events_end_dt_not_null::kshakir ran successfully in 3ms Running Changeset: changesets/sge_job_execution_unique_key.xml::sge_job_execution_unique_key::kshakir Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Unique constraint added to SGE_JOB(EXECUTION_ID) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/sge_job_execution_unique_key.xml::sge_job_execution_unique_key::kshakir ran successfully in 4ms Running Changeset: changesets/add_attempt_in_execution.xml::add-attempt-in-execution::tjeandet Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Columns ATTEMPT(INT) added to EXECUTION Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: NOT NULL constraint has been added to EXECUTION.ATTEMPT Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Foreign key FK_EXECUTION_WORKFLOW dropped Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Unique constraint UK_WORKFLOW_CALL_INDEX dropped from EXECUTION Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Unique constraint added to EXECUTION(WORKFLOW_EXECUTION_ID, CALL_FQN, IDX, ATTEMPT) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION (WORKFLOW_EXECUTION_ID) Jan 10, 2024 1:55:08 PM liquibase.changelog INFO: ChangeSet changesets/add_attempt_in_execution.xml::add-attempt-in-execution::tjeandet ran successfully in 18ms Running Changeset: changesets/execution_backend_info.xml::CREATE_EXECUTION_INFO::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table EXECUTION_INFO created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::CREATE_EXECUTION_INFO::mcovarr ran successfully in 5ms Running Changeset: changesets/execution_backend_info.xml::add_foreign_key_constraints::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION_INFO (EXECUTION_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::add_foreign_key_constraints::mcovarr ran successfully in 3ms Running Changeset: changesets/execution_backend_info.xml::add_backend_type_column::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Columns BACKEND_TYPE(VARCHAR(255)) added to EXECUTION Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::add_backend_type_column::mcovarr ran successfully in 3ms Running Changeset: changesets/execution_backend_info.xml::insert_data_from_backend_tables::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::insert_data_from_backend_tables::mcovarr ran successfully in 13ms Running Changeset: changesets/execution_backend_info.xml::set_backend_type::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::set_backend_type::mcovarr ran successfully in 9ms Running Changeset: changesets/execution_backend_info.xml::drop_job_tables::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table JES_JOB dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table SGE_JOB dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table LOCAL_JOB dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::drop_job_tables::mcovarr ran successfully in 11ms Running Changeset: changesets/execution_backend_info.xml::execution_info_uniqueness_constraint::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to EXECUTION_INFO(EXECUTION_ID, INFO_KEY) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::execution_info_uniqueness_constraint::mcovarr ran successfully in 3ms Running Changeset: changesets/execution_backend_info.xml::execution_backend_type_not_null::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: NOT NULL constraint has been added to EXECUTION.BACKEND_TYPE Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::execution_backend_type_not_null::mcovarr ran successfully in 3ms Running Changeset: changesets/runtime_attributes_table.xml::runtime-attributes-table::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table RUNTIME_ATTRIBUTES created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to RUNTIME_ATTRIBUTES (EXECUTION_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to RUNTIME_ATTRIBUTES(EXECUTION_ID, ATTRIBUTE_NAME) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/runtime_attributes_table.xml::runtime-attributes-table::tjeandet ran successfully in 8ms Running Changeset: changesets/failure_table.xml::failure_table::chrisl Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table FAILURE_EVENT created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to FAILURE_EVENT (EXECUTION_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to FAILURE_EVENT (WORKFLOW_EXECUTION_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/failure_table.xml::failure_table::chrisl ran successfully in 8ms Running Changeset: changesets/metadata_journal.xml::metadata_journal::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table METADATA_JOURNAL created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal.xml::metadata_journal::mcovarr ran successfully in 3ms Running Changeset: changesets/metadata_journal.xml::metadata_workflow_index::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index METADATA_WORKFLOW_IDX created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal.xml::metadata_workflow_index::mcovarr ran successfully in 2ms Running Changeset: changesets/metadata_journal.xml::metadata_job_index::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index METADATA_JOB_IDX created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal.xml::metadata_job_index::mcovarr ran successfully in 2ms Running Changeset: changesets/metadata_journal.xml::metadata_job_and_key_index::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index METADATA_JOB_AND_KEY_IDX created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal.xml::metadata_job_and_key_index::mcovarr ran successfully in 2ms Running Changeset: changesets/metadata_typed_values.xml::metadata_typed_values::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Columns METADATA_VALUE_TYPE(VARCHAR(10)) added to METADATA_JOURNAL Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/metadata_typed_values.xml::metadata_typed_values::tjeandet ran successfully in 4ms Running Changeset: changesets/workflow_metadata_summary.xml::metadata_journal_id_int_to_big_int::kshakir Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: METADATA_JOURNAL.METADATA_JOURNAL_ID datatype was changed to BIGINT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to METADATA_JOURNAL.METADATA_JOURNAL_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::metadata_journal_id_int_to_big_int::kshakir ran successfully in 5ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_summary::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_summary::mcovarr ran successfully in 3ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_uuid_idx::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_METADATA_SUMMARY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_uuid_idx::mcovarr ran successfully in 2ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_name_idx::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_NAME_IDX created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_name_idx::mcovarr ran successfully in 1ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_status_idx::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_STATUS_IDX created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_status_idx::mcovarr ran successfully in 1ms Running Changeset: changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_subsecond_timestamp::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: METADATA_JOURNAL.METADATA_TIMESTAMP datatype was changed to DATETIME(6) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_subsecond_timestamp::mcovarr ran successfully in 2ms Running Changeset: changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_timestamp_not_null::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: NOT NULL constraint has been added to METADATA_JOURNAL.METADATA_TIMESTAMP Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_timestamp_not_null::mcovarr ran successfully in 2ms Running Changeset: changesets/workflow_store.xml::WORKFLOW_STORE::chrisl Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table WORKFLOW_STORE created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store.xml::WORKFLOW_STORE::chrisl ran successfully in 4ms Running Changeset: changesets/workflow_store.xml::workflow_store_uuid_index::kshakir Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_STORE(WORKFLOW_UUID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store.xml::workflow_store_uuid_index::kshakir ran successfully in 2ms Running Changeset: changesets/workflow_store.xml::workflow_store_state_index::chrisl Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index WORKFLOW_STORE_STATE_IDX created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store.xml::workflow_store_state_index::chrisl ran successfully in 2ms Running Changeset: changesets/backend_KV_Store.xml::BACKEND_KV_STORE::rmunshi Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table BACKEND_KV_STORE created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/backend_KV_Store.xml::BACKEND_KV_STORE::rmunshi ran successfully in 3ms Running Changeset: changesets/backend_KV_Store.xml::backend_KV_store_job_key_constraint::rmunshi Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to BACKEND_KV_STORE(WORKFLOW_EXECUTION_UUID, CALL_FQN, JOB_SCATTER_INDEX, JOB_RETRY_ATTEMPT, STORE_KEY) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/backend_KV_Store.xml::backend_KV_store_job_key_constraint::rmunshi ran successfully in 3ms Running Changeset: changesets/job_store.xml::JOB_STORE::chrisl Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table JOB_STORE created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/job_store.xml::JOB_STORE::chrisl ran successfully in 3ms Running Changeset: changesets/job_store.xml::job_store_uuid_index::chrisl Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index JOB_STORE_UUID_IDX created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/job_store.xml::job_store_uuid_index::chrisl ran successfully in 1ms Running Changeset: changesets/job_store.xml::job_store_jobkey_index::chrisl Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE(WORKFLOW_UUID, CALL_FQN, JOB_SCATTER_INDEX, JOB_RETRY_ATTEMPT) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/job_store.xml::job_store_jobkey_index::chrisl ran successfully in 2ms Running Changeset: changesets/callcaching.xml::call_caching_result_metainfo::chrisl Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_METAINFO created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_metainfo::chrisl ran successfully in 2ms Running Changeset: changesets/callcaching.xml::call_caching_result_metainfo_uniqueness::chrisl Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_RESULT_METAINFO(WORKFLOW_UUID, CALL_FQN, JOB_SCATTER_INDEX) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_metainfo_uniqueness::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_hashes::chrisl Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table CALL_CACHING_HASH created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_hashes::chrisl ran successfully in 2ms Running Changeset: changesets/callcaching.xml::call_caching_result_hashes_uniqueness::chrisl Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_HASH(HASH_KEY, RESULT_METAINFO_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_hashes_uniqueness::chrisl ran successfully in 2ms Running Changeset: changesets/callcaching.xml::call_caching_result_hashes_uniqueness_foreign_key::chrisl Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH (RESULT_METAINFO_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_hashes_uniqueness_foreign_key::chrisl ran successfully in 2ms Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons::chrisl Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_SIMPLETON created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_simpletons::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons_uniqueness::chrisl Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_RESULT_SIMPLETON(SIMPLETON_KEY, RESULT_METAINFO_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_simpletons_uniqueness::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons_foreign_key::chrisl Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_RESULT_SIMPLETON (RESULT_METAINFO_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_simpletons_foreign_key::chrisl ran successfully in 2ms Running Changeset: changesets/call_caching_allow_result_reuse_fix.xml::call_caching_fix_boolean::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: CALL_CACHING_RESULT_METAINFO.ALLOW_RESULT_REUSE datatype was changed to BOOLEAN Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_allow_result_reuse_fix.xml::call_caching_fix_boolean::mcovarr ran successfully in 1ms Running Changeset: changesets/call_caching_allow_result_reuse_fix.xml::call_caching_reuse_default::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Default value added to CALL_CACHING_RESULT_METAINFO.ALLOW_RESULT_REUSE Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_allow_result_reuse_fix.xml::call_caching_reuse_default::mcovarr ran successfully in 2ms Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table JOB_STORE_RESULT_SIMPLETON created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_result_simpletons::mcovarr ran successfully in 2ms Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons_uniqueness::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE_RESULT_SIMPLETON(SIMPLETON_KEY, JOB_STORE_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_result_simpletons_uniqueness::mcovarr ran successfully in 2ms Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons_foreign_key::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_RESULT_SIMPLETON (JOB_STORE_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_result_simpletons_foreign_key::mcovarr ran successfully in 2ms Running Changeset: changesets/job_store_simpletons.xml::job_store_remove_job_output::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column JOB_STORE.JOB_OUTPUT dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_remove_job_output::mcovarr ran successfully in 2ms Running Changeset: changesets/restart_and_recover_migration.xml::WORKFLOW_STORE_MIGRATION::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::WORKFLOW_STORE_MIGRATION::mcovarr ran successfully in 4ms Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_MIGRATION::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::JOB_STORE_MIGRATION::mcovarr ran successfully in 6ms Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_VALUE_NULLABLE::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Null constraint dropped from JOB_STORE_RESULT_SIMPLETON.SIMPLETON_VALUE Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_VALUE_NULLABLE::mcovarr ran successfully in 17ms Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_MIGRATION::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: WORKFLOW_EXECUTION + EXECUTION + SYMBOL + JOB_STORE -> JOB_STORE_RESULT_SIMPLETON migration complete. Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_MIGRATION::mcovarr ran successfully in 2ms Running Changeset: changesets/restart_and_recover_migration.xml::BACKEND_KV_STORE_MIGRATION::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::BACKEND_KV_STORE_MIGRATION::mcovarr ran successfully in 3ms Running Changeset: changesets/restart_and_recover_migration.xml::WORKFLOW_OPTIONS_RENAME_MIGRATION::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Workflow option renaming migration complete. Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::WORKFLOW_OPTIONS_RENAME_MIGRATION::mcovarr ran successfully in 2ms Running Changeset: changesets/summary_status_table.xml::summary_status_table::kshakir Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table SUMMARY_STATUS created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/summary_status_table.xml::summary_status_table::kshakir ran successfully in 3ms Running Changeset: changesets/summary_status_table.xml::summary_status_summarized_table_name_index::kshakir Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS(SUMMARY_TABLE_NAME, SUMMARIZED_TABLE_NAME) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/summary_status_table.xml::summary_status_summarized_table_name_index::kshakir ran successfully in 2ms Running Changeset: changesets/standardize_column_names.xml::workflow_execution_uuid::rmunshi Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column CALL_CACHING_RESULT_METAINFO.WORKFLOW_UUID renamed to WORKFLOW_EXECUTION_UUID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column WORKFLOW_STORE.WORKFLOW_UUID renamed to WORKFLOW_EXECUTION_UUID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column JOB_STORE.WORKFLOW_UUID renamed to WORKFLOW_EXECUTION_UUID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names.xml::workflow_execution_uuid::rmunshi ran successfully in 3ms Running Changeset: changesets/standardize_column_names.xml::call_job_identifiers::rmunshi Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column METADATA_JOURNAL.METADATA_CALL_FQN renamed to CALL_FQN Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column METADATA_JOURNAL.METADATA_CALL_INDEX renamed to JOB_SCATTER_INDEX Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column METADATA_JOURNAL.METADATA_CALL_ATTEMPT renamed to JOB_RETRY_ATTEMPT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names.xml::call_job_identifiers::rmunshi ran successfully in 4ms Running Changeset: changesets/embiggen_metadata_value.xml::entry_or_journal_existence_xor::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/embiggen_metadata_value.xml::entry_or_journal_existence_xor::mcovarr ran successfully in 14ms Running Changeset: changesets/embiggen_metadata_value.xml::embiggen_metadata_entry::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Marking ChangeSet: changesets/embiggen_metadata_value.xml::embiggen_metadata_entry::mcovarr ran despite precondition failure due to onFail='MARK_RAN': changelog.xml : Table PUBLIC.METADATA_ENTRY does not exist Running Changeset: changesets/embiggen_metadata_value.xml::embiggen_metadata_journal::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: METADATA_JOURNAL.METADATA_VALUE datatype was changed to LONGTEXT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/embiggen_metadata_value.xml::embiggen_metadata_journal::mcovarr ran successfully in 4ms Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus::rmunshi Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table CALL_CACHING_JOB_DETRITUS created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_job_detritus.xml::call_caching_job_detritus::rmunshi ran successfully in 2ms Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus_key_uniqueness::rmunshi Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_JOB_DETRITUS(JOB_DETRITUS_KEY, RESULT_METAINFO_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_job_detritus.xml::call_caching_job_detritus_key_uniqueness::rmunshi ran successfully in 2ms Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus_foreign_key::rmunshi Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_JOB_DETRITUS (RESULT_METAINFO_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_job_detritus.xml::call_caching_job_detritus_foreign_key::rmunshi ran successfully in 2ms Running Changeset: changesets/job_store_tinyints.xml::job_store_fix_job_successful::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: JOB_STORE.JOB_SUCCESSFUL datatype was changed to BOOLEAN Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/job_store_tinyints.xml::job_store_fix_job_successful::mcovarr ran successfully in 1ms Running Changeset: changesets/job_store_tinyints.xml::job_store_fix_job_retryable_failure::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: JOB_STORE.RETRYABLE_FAILURE datatype was changed to BOOLEAN Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/job_store_tinyints.xml::job_store_fix_job_retryable_failure::mcovarr ran successfully in 1ms Running Changeset: changesets/job_store_tinyints.xml::job_store_not_nullable_job_successful::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: NOT NULL constraint has been added to JOB_STORE.JOB_SUCCESSFUL Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/job_store_tinyints.xml::job_store_not_nullable_job_successful::mcovarr ran successfully in 1ms Running Changeset: changesets/standardize_column_names_again.xml::standardize_column_names_again::kshakir Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index JOB_STORE_UUID_IDX dropped from table JOB_STORE Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_NAME_IDX dropped from table WORKFLOW_METADATA_SUMMARY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_STATUS_IDX dropped from table WORKFLOW_METADATA_SUMMARY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index WORKFLOW_STORE_STATE_IDX dropped from table WORKFLOW_STORE Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_JOB_DETRITUS_KEY dropped from CALL_CACHING_JOB_DETRITUS Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_RESULT_METAINFO dropped from CALL_CACHING_RESULT_METAINFO Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_HASH dropped from CALL_CACHING_HASH Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_RESULT_SIMPLETON dropped from CALL_CACHING_RESULT_SIMPLETON Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint JOB_STORE_JOBKEY_IDX dropped from JOB_STORE Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint UK_BACKEND_KV_STORE_KEY dropped from BACKEND_KV_STORE Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint UK_JOB_STORE_RESULT_SIMPLETON dropped from JOB_STORE_RESULT_SIMPLETON Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint SUMMARY_STATUS_SUMMARY_TABLE_NAME_SUMMARIZED_TABLE_NAME_INDEX dropped from SUMMARY_STATUS Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint WORKFLOW_METADATA_UUID_IDX dropped from WORKFLOW_METADATA_SUMMARY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint WORKFLOW_STORE_UUID_IDX dropped from WORKFLOW_STORE Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key CCJD_RESULT_METAINFO_ID_FK dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key CCH_RESULT_METAINFO_ID_FK dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key CCRS_RESULT_METAINFO_ID_FK dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key JSRS_JOB_STORE_FK dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table CALL_CACHING_JOB_DETRITUS renamed to CALL_CACHING_DETRITUS_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_METAINFO renamed to CALL_CACHING_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table CALL_CACHING_HASH renamed to CALL_CACHING_HASH_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_SIMPLETON renamed to CALL_CACHING_SIMPLETON_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table BACKEND_KV_STORE renamed to JOB_KEY_VALUE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table JOB_STORE renamed to JOB_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table JOB_STORE_RESULT_SIMPLETON renamed to JOB_STORE_SIMPLETON_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table METADATA_JOURNAL renamed to METADATA_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table SUMMARY_STATUS renamed to SUMMARY_STATUS_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY renamed to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table WORKFLOW_STORE renamed to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_JOB_DETRITUS_ID renamed to CALL_CACHING_DETRITUS_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.JOB_DETRITUS_KEY renamed to DETRITUS_KEY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.JOB_DETRITUS_VALUE renamed to DETRITUS_VALUE Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column CALL_CACHING_ENTRY.CALL_CACHING_RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column CALL_CACHING_ENTRY.JOB_SCATTER_INDEX renamed to JOB_INDEX Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column CALL_CACHING_ENTRY.CALL_FQN renamed to CALL_FULLY_QUALIFIED_NAME Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column CALL_CACHING_HASH_ENTRY.RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ID renamed to CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column CALL_CACHING_SIMPLETON_ENTRY.RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_RESULT_SIMPLETON_ID renamed to CALL_CACHING_SIMPLETON_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.JOB_RETRY_ATTEMPT renamed to JOB_ATTEMPT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.JOB_SCATTER_INDEX renamed to JOB_INDEX Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.BACKEND_KV_STORE_ID renamed to JOB_KEY_VALUE_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.CALL_FQN renamed to CALL_FULLY_QUALIFIED_NAME Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.JOB_RETRY_ATTEMPT renamed to JOB_ATTEMPT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.JOB_SCATTER_INDEX renamed to JOB_INDEX Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.JOB_STORE_ID renamed to JOB_STORE_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.CALL_FQN renamed to CALL_FULLY_QUALIFIED_NAME Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_ID renamed to JOB_STORE_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_RESULT_SIMPLETON_ID renamed to JOB_STORE_SIMPLETON_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.SUMMARY_STATUS_ID renamed to SUMMARY_STATUS_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column WORKFLOW_METADATA_SUMMARY_ENTRY.END_DT renamed to END_TIMESTAMP Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column WORKFLOW_METADATA_SUMMARY_ENTRY.START_DT renamed to START_TIMESTAMP Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column WORKFLOW_METADATA_SUMMARY_ENTRY.WORKFLOW_METADATA_SUMMARY_ID renamed to WORKFLOW_METADATA_SUMMARY_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column WORKFLOW_STORE_ENTRY.WORKFLOW_STORE_ID renamed to WORKFLOW_STORE_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_DETRITUS_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_HASH_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_SIMPLETON_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key dropped from JOB_KEY_VALUE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key dropped from JOB_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key dropped from JOB_STORE_SIMPLETON_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key dropped from SUMMARY_STATUS_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key dropped from WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key dropped from WORKFLOW_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_DETRITUS_ENTRY (CALL_CACHING_DETRITUS_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_HASH_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_SIMPLETON_ENTRY (CALL_CACHING_SIMPLETON_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key added to JOB_KEY_VALUE_ENTRY (JOB_KEY_VALUE_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key added to JOB_STORE_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_SIMPLETON_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key added to SUMMARY_STATUS_ENTRY (SUMMARY_STATUS_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key added to WORKFLOW_METADATA_SUMMARY_ENTRY (WORKFLOW_METADATA_SUMMARY_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Primary key added to WORKFLOW_STORE_ENTRY (WORKFLOW_STORE_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_DETRITUS_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_ENTRY.CALL_CACHING_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_SIMPLETON_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to JOB_KEY_VALUE_ENTRY.JOB_KEY_VALUE_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_ENTRY.JOB_STORE_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_SIMPLETON_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to SUMMARY_STATUS_ENTRY.SUMMARY_STATUS_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to WORKFLOW_METADATA_SUMMARY_ENTRY.WORKFLOW_METADATA_SUMMARY_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to WORKFLOW_STORE_ENTRY.WORKFLOW_STORE_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_DETRITUS_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_SIMPLETON_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_DETRITUS_ENTRY(CALL_CACHING_ENTRY_ID, DETRITUS_KEY) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_HASH_ENTRY(CALL_CACHING_ENTRY_ID, HASH_KEY) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_SIMPLETON_ENTRY(CALL_CACHING_ENTRY_ID, SIMPLETON_KEY) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to JOB_KEY_VALUE_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX, JOB_ATTEMPT, STORE_KEY) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX, JOB_ATTEMPT) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE_SIMPLETON_ENTRY(JOB_STORE_ENTRY_ID, SIMPLETON_KEY) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS_ENTRY(SUMMARY_TABLE_NAME, SUMMARIZED_TABLE_NAME) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_METADATA_SUMMARY_ENTRY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_STORE_ENTRY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index IX_JOB_STORE_ENTRY_WEU created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WN created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WS created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index IX_WORKFLOW_STORE_ENTRY_WS created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_again.xml::standardize_column_names_again::kshakir ran successfully in 139ms Running Changeset: changesets/rename_workflow_options_in_metadata.xml::rename_workflow_options_in_metadata::tjeandet [2024-01-10 13:55:09,56] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000 [2024-01-10 13:55:09,57] [info] [RenameWorkflowOptionsInMetadata] 100% Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: RenameWorkflowOptionsInMetadata complete. Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/rename_workflow_options_in_metadata.xml::rename_workflow_options_in_metadata::tjeandet ran successfully in 19ms Running Changeset: changesets/encrypt_and_clear_workflow_options.xml::encrypt_workflow_store_entry_workflow_options::kshakir Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: EncryptWorkflowStoreEntryWorkflowOptions complete. Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/encrypt_and_clear_workflow_options.xml::encrypt_workflow_store_entry_workflow_options::kshakir ran successfully in 2ms Running Changeset: changesets/encrypt_and_clear_workflow_options.xml::clear_metadata_entry_workflow_options::kshakir Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ClearMetadataEntryWorkflowOptions complete. Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/encrypt_and_clear_workflow_options.xml::clear_metadata_entry_workflow_options::kshakir ran successfully in 1ms Running Changeset: changesets/sub_workflow_store.xml::SUB_WORKFLOW_STORE_ENTRY::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table SUB_WORKFLOW_STORE_ENTRY created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::SUB_WORKFLOW_STORE_ENTRY::tjeandet ran successfully in 2ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_uuid_index::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index IX_SUB_WORKFLOW_STORE_ENTRY_PWEU created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_uuid_index::tjeandet ran successfully in 2ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_jobkey_index::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to SUB_WORKFLOW_STORE_ENTRY(PARENT_WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, CALL_INDEX, CALL_ATTEMPT) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_jobkey_index::tjeandet ran successfully in 1ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_root_workflow_fk::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to SUB_WORKFLOW_STORE_ENTRY (ROOT_WORKFLOW_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_root_workflow_fk::tjeandet ran successfully in 2ms Running Changeset: changesets/workflow_store_imports_file.xml::workflow-store-imports-file::cjllanwarne Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Columns IMPORTS_ZIP(BLOB) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_imports_file.xml::workflow-store-imports-file::cjllanwarne ran successfully in 2ms Running Changeset: changesets/workflow_store_labels_file.xml::workflow-store-labels-file::cjllanwarne Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Columns CUSTOM_LABELS(LONGVARCHAR) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_labels_file.xml::workflow-store-labels-file::cjllanwarne ran successfully in 2ms Running Changeset: changesets/embiggen_detritus_value.xml::embiggen_call_caching_detritus_entry::rmunshi Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE datatype was changed to LONGVARCHAR Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/embiggen_detritus_value.xml::embiggen_call_caching_detritus_entry::rmunshi ran successfully in 2ms Running Changeset: changesets/standardize_column_names_patches.xml::standardize_column_names_patches::kshakir Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_HASH_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key FK_JOB_STORE_RESULT_SIMPLETON_JOB_STORE_ENTRY_ID dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key FK_SUB_WORKFLOW_STORE_ROOT_WORKFLOW_ID_WORKFLOW_STORE_ENTRY_ID dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint UC_CALL_CACHING_ENTRY_WEU_CQFN_JI dropped from CALL_CACHING_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint UC_CALL_CACHING_HASH_ENTRY_CCEI dropped from CALL_CACHING_HASH_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_HASH_ENTRY(CALL_CACHING_ENTRY_ID, HASH_KEY) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to SUB_WORKFLOW_STORE_ENTRY (ROOT_WORKFLOW_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_patches.xml::standardize_column_names_patches::kshakir ran successfully in 14ms Running Changeset: changesets/standardize_column_names_patches.xml::hsqldb_longvarchar_to_longtext::kshakir Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE datatype was changed to LONGTEXT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.SIMPLETON_VALUE datatype was changed to LONGTEXT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: JOB_STORE_ENTRY.EXCEPTION_MESSAGE datatype was changed to LONGTEXT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: JOB_STORE_SIMPLETON_ENTRY.SIMPLETON_VALUE datatype was changed to LONGTEXT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.CUSTOM_LABELS datatype was changed to LONGTEXT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_DEFINITION datatype was changed to LONGTEXT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_INPUTS datatype was changed to LONGTEXT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_OPTIONS datatype was changed to LONGTEXT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_patches.xml::hsqldb_longvarchar_to_longtext::kshakir ran successfully in 11ms Running Changeset: changesets/nullable_lobs.xml::nullable_lobs::kshakir Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Null constraint dropped from CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Null constraint dropped from CALL_CACHING_SIMPLETON_ENTRY.SIMPLETON_VALUE Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Null constraint dropped from JOB_STORE_SIMPLETON_ENTRY.SIMPLETON_VALUE Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_DEFINITION Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_INPUTS Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_OPTIONS Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.CUSTOM_LABELS Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/nullable_lobs.xml::nullable_lobs::kshakir ran successfully in 7ms Running Changeset: changesets/add_attempt_in_call_caching_entry.xml::add_attempt_in_call_caching_entry::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Columns JOB_ATTEMPT(INT) added to CALL_CACHING_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/add_attempt_in_call_caching_entry.xml::add_attempt_in_call_caching_entry::tjeandet ran successfully in 2ms Running Changeset: changesets/replace_empty_custom_labels.xml::custom_labels_not_null::rmunshi Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_STORE_ENTRY.CUSTOM_LABELS Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/replace_empty_custom_labels.xml::custom_labels_not_null::rmunshi ran successfully in 3ms Running Changeset: changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table CALL_CACHING_AGGREGATION_ENTRY created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry::tjeandet ran successfully in 2ms Running Changeset: changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry_keys_and_indexes::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_AGGREGATION_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Index IX_CALL_CACHING_AGGREGATION_ENTRY_BA_IFA created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry_keys_and_indexes::tjeandet ran successfully in 3ms Running Changeset: changesets/custom_label_entry.xml::custom_LABEL_entry::rmunshi Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::custom_LABEL_entry::rmunshi ran successfully in 2ms Running Changeset: changesets/custom_label_entry.xml::WMSE_workflow_execution_uuid_foreign_key::rmunshi Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to CUSTOM_LABEL_ENTRY (WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::WMSE_workflow_execution_uuid_foreign_key::rmunshi ran successfully in 1ms Running Changeset: changesets/custom_label_entry.xml::custom_label_entry_index::rmunshi Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, CUSTOM_LABEL_VALUE, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::custom_label_entry_index::rmunshi ran successfully in 2ms Running Changeset: changesets/custom_label_entry.xml::modify label key/value data types::rmunshi Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_KEY datatype was changed to VARCHAR(63) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_VALUE datatype was changed to VARCHAR(63) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::modify label key/value data types::rmunshi ran successfully in 3ms Running Changeset: changesets/docker_hash_store.xml::DOCKER_HASH_STORE_ENTRY::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table DOCKER_HASH_STORE_ENTRY created Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store.xml::DOCKER_HASH_STORE_ENTRY::tjeandet ran successfully in 3ms Running Changeset: changesets/docker_hash_store.xml::docker_hash_store_uuid_unique_constraint::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint added to DOCKER_HASH_STORE_ENTRY(WORKFLOW_EXECUTION_UUID, DOCKER_TAG) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store.xml::docker_hash_store_uuid_unique_constraint::tjeandet ran successfully in 2ms Running Changeset: changesets/workflow_store_type_and_version.xml::workflow-store-type-and-version::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Columns WORKFLOW_TYPE(VARCHAR(30)),WORKFLOW_TYPE_VERSION(VARCHAR(255)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_type_and_version.xml::workflow-store-type-and-version::mcovarr ran successfully in 3ms Running Changeset: changesets/remove_pre_pbe_tables.xml::drop-pre-pbe-tables::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table EXECUTION_INFO dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table EXECUTION_EVENT dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table FAILURE_EVENT dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table RUNTIME_ATTRIBUTES dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table SYMBOL dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table EXECUTION dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table WORKFLOW_EXECUTION_AUX dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table WORKFLOW_EXECUTION dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/remove_pre_pbe_tables.xml::drop-pre-pbe-tables::mcovarr ran successfully in 14ms Running Changeset: changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key FK_CUSTOM_LABEL_ENTRY_WORKFLOW_EXECUTION_UUID dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint UC_CUSTOM_LABEL_ENTRY_CLK_CLV_WEU dropped from CUSTOM_LABEL_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint UC_SUMMARY_STATUS_ENTRY_STN_STN dropped from SUMMARY_STATUS_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Unique constraint UC_WORKFLOW_METADATA_SUMMARY_ENTRY_WEU dropped from WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table METADATA_ENTRY dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table SUMMARY_STATUS_ENTRY dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY_ENTRY dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir ran successfully in 10ms Running Changeset: changesets/workflow_store_state_widening.xml::workflow-store-state-widening::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_STATE datatype was changed to varchar(20) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_state_widening.xml::workflow-store-state-widening::tjeandet ran successfully in 1ms Running Changeset: changesets/workflow_store_restarted_column.xml::workflow-store-restarted-column::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Columns RESTARTED(BOOLEAN) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_restarted_column.xml::workflow-store-restarted-column::tjeandet ran successfully in 4ms Running Changeset: changesets/workflow_store_restarted_column.xml::update-restartable::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Data updated in WORKFLOW_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Data updated in WORKFLOW_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_restarted_column.xml::update-restartable::tjeandet ran successfully in 3ms Running Changeset: changesets/workflow_store_workflow_root_column.xml::workflow-store-workflow-root-column::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Columns WORKFLOW_ROOT(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_workflow_root_column.xml::workflow-store-workflow-root-column::tjeandet ran successfully in 2ms Running Changeset: changesets/workflow_store_horizontal_db.xml::workflow-store-horizontal-db::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Column WORKFLOW_STORE_ENTRY.RESTARTED dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Columns CROMWELL_ID(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Columns HEARTBEAT_TIMESTAMP(TIMESTAMP) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_horizontal_db.xml::workflow-store-horizontal-db::mcovarr ran successfully in 5ms Running Changeset: changesets/add_workflow_url_in_workflow_store_entry.xml::add_workflow_url_in_workflow_store_entry::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Columns WORKFLOW_URL(VARCHAR(255)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/add_workflow_url_in_workflow_store_entry.xml::add_workflow_url_in_workflow_store_entry::sshah ran successfully in 2ms Running Changeset: changesets/change_max_size_workflow_url.xml::change_max_size_for_workflow_url::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_URL datatype was changed to VARCHAR(2000) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/change_max_size_workflow_url.xml::change_max_size_for_workflow_url::sshah ran successfully in 1ms Running Changeset: changesets/docker_hash_store_add_size_column.xml::docker_hash_store_entry_add_size_column::tjeandet Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Columns DOCKER_SIZE(BIGINT) added to DOCKER_HASH_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store_add_size_column.xml::docker_hash_store_entry_add_size_column::tjeandet ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_hash_entry_id.xml::restore_auto_increment_call_caching_hash_entry_id::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_hash_entry_id.xml::restore_auto_increment_call_caching_hash_entry_id::mcovarr ran successfully in 2ms Running Changeset: changesets/add_hog_group_in_workflow_store.xml::add_hog_group_in_workflow_store::cjllanwarne Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Columns HOG_GROUP(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/add_hog_group_in_workflow_store.xml::add_hog_group_in_workflow_store::cjllanwarne ran successfully in 2ms Running Changeset: changesets/resync_engine_schema.xml::resync-engine-schema::kshakir Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.HEARTBEAT_TIMESTAMP datatype was changed to DATETIME(6) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.SUBMISSION_TIME datatype was changed to DATETIME(6) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_STORE_ENTRY.SUBMISSION_TIME Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/resync_engine_schema.xml::resync-engine-schema::kshakir ran successfully in 4ms Running Changeset: changesets/enlarge_job_store_ids.xml::drop_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key FK_JOB_STORE_SIMPLETON_ENTRY_JOB_STORE_ENTRY_ID dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::drop_job_store_simpleton_entry_fk::mcovarr ran successfully in 1ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_entry_id::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_ENTRY.JOB_STORE_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_entry_id::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_id::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_SIMPLETON_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_id::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_fk::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::recreate_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::recreate_job_store_simpleton_entry_fk::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_AGGREGATION_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_detritus_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_DETRITUS_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_detritus_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_hash_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_HASH_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_hash_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_SIMPLETON_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_cache_entry_id::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: CALL_CACHING_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_cache_entry_id::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::reset_call_cache_entry_id_autoincrement::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_ENTRY.CALL_CACHING_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::reset_call_cache_entry_id_autoincrement::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_aggregation_entry_fk::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_aggregation_entry_fk::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_detritus_entry_fk::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_detritus_entry_fk::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_hash_entry_fk::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: CALL_CACHING_HASH_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_hash_entry_fk::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_simpleton_entry_fk::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_simpleton_entry_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_AGGREGATION_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_detritus_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_DETRITUS_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_detritus_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_hash_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_hash_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_SIMPLETON_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_aggregation_entry_id.xml::enlarge_call_cache_aggregation_entry_id::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_AGGREGATION_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_aggregation_entry_id.xml::enlarge_call_cache_aggregation_entry_id::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_aggregation_entry_id.xml::reset_call_caching_aggregation_entry_id_autoincrement::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_AGGREGATION_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_aggregation_entry_id.xml::reset_call_caching_aggregation_entry_id_autoincrement::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_detritus_entry_id.xml::enlarge_call_cache_detritus_entry_id::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_DETRITUS_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_detritus_entry_id.xml::enlarge_call_cache_detritus_entry_id::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_detritus_entry_id.xml::reset_call_caching_detritus_entry_id_autoincrement::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_DETRITUS_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_detritus_entry_id.xml::reset_call_caching_detritus_entry_id_autoincrement::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_simpleton_entry_id.xml::enlarge_call_cache_simpleton_entry_id::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_SIMPLETON_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_simpleton_entry_id.xml::enlarge_call_cache_simpleton_entry_id::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_simpleton_entry_id.xml::reset_call_caching_simpleton_entry_id_autoincrement::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_SIMPLETON_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_simpleton_entry_id.xml::reset_call_caching_simpleton_entry_id_autoincrement::sshah ran successfully in 2ms Running Changeset: changesets/reset_call_caching_hash_entry_id_autoincrement.xml::reset_call_caching_hash_entry_id_autoincrement::sshah Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 1:55:09 PM liquibase.changelog INFO: ChangeSet changesets/reset_call_caching_hash_entry_id_autoincrement.xml::reset_call_caching_hash_entry_id_autoincrement::sshah ran successfully in 1ms Jan 10, 2024 1:55:09 PM liquibase.lockservice INFO: Successfully released change log lock [2024-01-10 13:55:09,97] [info] Running with database db.url = jdbc:hsqldb:mem:7d11527d-aa1f-4017-870b-8b0c4a215932;shutdown=false;hsqldb.tx=mvcc Jan 10, 2024 1:55:10 PM liquibase.database INFO: Set default schema name to PUBLIC Jan 10, 2024 1:55:10 PM liquibase.lockservice INFO: Successfully acquired change log lock Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Creating database history table with name: PUBLIC.SQLMETADATADATABASECHANGELOG Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Reading from PUBLIC.SQLMETADATADATABASECHANGELOG Running Changeset: metadata_changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Table METADATA_ENTRY created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Table SUMMARY_STATUS_ENTRY created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY_ENTRY created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, CUSTOM_LABEL_VALUE, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS_ENTRY(SUMMARY_TABLE_NAME, SUMMARIZED_TABLE_NAME) Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_METADATA_SUMMARY_ENTRY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Foreign key constraint added to CUSTOM_LABEL_ENTRY (WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WN created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WS created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Index METADATA_JOB_AND_KEY_IDX created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Index METADATA_JOB_IDX created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Index METADATA_WORKFLOW_IDX created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Index SYS_IDX_11226 created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir ran successfully in 22ms Running Changeset: changesets/change_max_size_label_entry.xml::change_max_size_for_key_and_value_in_label::sshah Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_KEY datatype was changed to VARCHAR(255) Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_VALUE datatype was changed to VARCHAR(255) Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet changesets/change_max_size_label_entry.xml::change_max_size_for_key_and_value_in_label::sshah ran successfully in 3ms Running Changeset: metadata_changesets/delete_duplicate_custom_labels.xml::delete_duplicate_custom_labels::kshakir Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Data deleted from CUSTOM_LABEL_ENTRY Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Unique constraint UC_CUSTOM_LABEL_ENTRY_CLK_CLV_WEU dropped from CUSTOM_LABEL_ENTRY Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/delete_duplicate_custom_labels.xml::delete_duplicate_custom_labels::kshakir ran successfully in 6ms Running Changeset: metadata_changesets/add_submission_timestamp_metadata_summary.xml::add_submission_timestamp_in_metadata_entry::sshah Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Columns SUBMISSION_TIMESTAMP(DATETIME(6)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_submission_timestamp_metadata_summary.xml::add_submission_timestamp_in_metadata_entry::sshah ran successfully in 2ms Running Changeset: metadata_changesets/custom_label_entry_key_value_index.xml::custom_label_entry_key_value_index::mcovarr Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Index IX_CUSTOM_LABEL_ENTRY_CLK_CLV created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/custom_label_entry_key_value_index.xml::custom_label_entry_key_value_index::mcovarr ran successfully in 2ms Running Changeset: metadata_changesets/add_parent_and_root_workflow_execution_uuid.xml::add_parent_and_root_workflow_execution_uuid::kshakir Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Columns PARENT_WORKFLOW_EXECUTION_UUID(VARCHAR(100)),ROOT_WORKFLOW_EXECUTION_UUID(VARCHAR(100)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_PWEU created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_RWEU created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Unique constraint UC_SUMMARY_STATUS_ENTRY_STN_STN dropped from SUMMARY_STATUS_ENTRY Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.SUMMARY_TABLE_NAME renamed to SUMMARY_NAME Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.SUMMARIZED_TABLE_NAME dropped Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.MAXIMUM_ID renamed to SUMMARY_POSITION Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS_ENTRY(SUMMARY_NAME) Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_parent_and_root_workflow_execution_uuid.xml::add_parent_and_root_workflow_execution_uuid::kshakir ran successfully in 11ms Running Changeset: metadata_changesets/add_parent_and_root_workflow_typo_fix.xml::add_parent_and_root_workflow_typo_fix::kshakir Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Data deleted from SUMMARY_STATUS_ENTRY Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_parent_and_root_workflow_typo_fix.xml::add_parent_and_root_workflow_typo_fix::kshakir ran successfully in 2ms Running Changeset: metadata_changesets/metadata_index_removals.xml::metadata_index_removals::mcovarr Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Index METADATA_JOB_AND_KEY_IDX dropped from table METADATA_ENTRY Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Index METADATA_JOB_IDX dropped from table METADATA_ENTRY Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/metadata_index_removals.xml::metadata_index_removals::mcovarr ran successfully in 2ms Running Changeset: metadata_changesets/add_metadata_archive_status.xml::add_metadata_archive_status::cjllanwarne Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Columns METADATA_ARCHIVE_STATUS(VARCHAR(30)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_metadata_archive_status.xml::add_metadata_archive_status::cjllanwarne ran successfully in 2ms Running Changeset: metadata_changesets/add_metadata_archive_status.xml::metadata_archive_status_index::cjllanwarne Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_metadata_archive_status.xml::metadata_archive_status_index::cjllanwarne ran successfully in 1ms Running Changeset: metadata_changesets/summarization_queue_table.xml::summary_queue_table::gsterin Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Table SUMMARY_QUEUE_ENTRY created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table.xml::summary_queue_table::gsterin ran successfully in 2ms Running Changeset: metadata_changesets/summarization_queue_table.xml::summary_queue_initialization::gsterin Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table.xml::summary_queue_initialization::gsterin ran successfully in 6ms Running Changeset: metadata_changesets/summarization_queue_table_add_primary_key.xml::summary_queue_add_primary_key::gsterin Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Primary key added to SUMMARY_QUEUE_ENTRY (METADATA_JOURNAL_ID) Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table_add_primary_key.xml::summary_queue_add_primary_key::gsterin ran successfully in 2ms Running Changeset: metadata_changesets/remove_non_summarizable_metadata_from_queue.xml::delete_non_summarizable_metadata_from_queue::mcovarr Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/remove_non_summarizable_metadata_from_queue.xml::delete_non_summarizable_metadata_from_queue::mcovarr ran successfully in 3ms Running Changeset: metadata_changesets/update_metadata_archive_index.xml::update_old_metadata_archive_status_index::cjllanwarne Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS dropped from table WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS_ET created Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/update_metadata_archive_index.xml::update_old_metadata_archive_status_index::cjllanwarne ran successfully in 2ms Running Changeset: metadata_changesets/reset_archive_statuses_to_null.xml::reset_archive_statuses_to_null::cjllanwarne Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: Data updated in WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:55:10 PM liquibase.changelog INFO: ChangeSet metadata_changesets/reset_archive_statuses_to_null.xml::reset_archive_statuses_to_null::cjllanwarne ran successfully in 2ms Jan 10, 2024 1:55:10 PM liquibase.lockservice INFO: Successfully released change log lock [2024-01-10 13:55:10,55] [info] Slf4jLogger started [2024-01-10 13:55:10,81] [info] Workflow heartbeat configuration: { "cromwellId" : "cromid-01e5e21", "heartbeatInterval" : "2 minutes", "ttl" : "10 minutes", "failureShutdownDuration" : "5 minutes", "writeBatchSize" : 10000, "writeThreshold" : 10000 } [2024-01-10 13:55:10,90] [info] Metadata summary refreshing every 1 second. [2024-01-10 13:55:10,90] [info] No metadata archiver defined in config [2024-01-10 13:55:10,90] [info] No metadata deleter defined in config [2024-01-10 13:55:10,93] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds. [2024-01-10 13:55:10,93] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds. [2024-01-10 13:55:10,96] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds. [2024-01-10 13:55:11,05] [info] JobRestartCheckTokenDispenser - Distribution rate: 50 per 1 seconds. [2024-01-10 13:55:11,09] [info] JobExecutionTokenDispenser - Distribution rate: 20 per 10 seconds. [2024-01-10 13:55:11,14] [info] SingleWorkflowRunnerActor: Version 83 [2024-01-10 13:55:11,15] [info] SingleWorkflowRunnerActor: Submitting workflow [2024-01-10 13:55:11,23] [info] Unspecified type (Unspecified version) workflow 6d31935b-042f-4566-b5b7-a155b681d22c submitted [2024-01-10 13:55:11,26] [info] SingleWorkflowRunnerActor: Workflow submitted 6d31935b-042f-4566-b5b7-a155b681d22c [2024-01-10 13:55:11,26] [info] 1 new workflows fetched by cromid-01e5e21: 6d31935b-042f-4566-b5b7-a155b681d22c [2024-01-10 13:55:11,27] [info] WorkflowManagerActor: Starting workflow 6d31935b-042f-4566-b5b7-a155b681d22c [2024-01-10 13:55:11,28] [info] WorkflowManagerActor: Successfully started WorkflowActor-6d31935b-042f-4566-b5b7-a155b681d22c [2024-01-10 13:55:11,28] [info] Retrieved 1 workflows from the WorkflowStoreActor [2024-01-10 13:55:11,32] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes. [2024-01-10 13:55:11,44] [info] MaterializeWorkflowDescriptorActor [6d31935b]: Parsing workflow as WDL 1.0 [2024-01-10 13:55:13,01] [info] MaterializeWorkflowDescriptorActor [6d31935b]: Call-to-Backend assignments: theiacov_fasta.consensus_qc -> Local, theiacov_fasta.vadr -> Local, theiacov_fasta.version_capture -> Local, theiacov_fasta.nextclade_output_parser -> Local, theiacov_fasta.nextclade -> Local, theiacov_fasta.qc_check_phb -> Local, theiacov_fasta.abricate_flu -> Local, theiacov_fasta.pangolin4 -> Local [2024-01-10 13:55:13,23] [warn] Local [6d31935b]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:55:13,23] [warn] Local [6d31935b]: Key/s [dx_instance_type, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:55:13,24] [warn] Local [6d31935b]: Key/s [dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:55:13,24] [warn] Local [6d31935b]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:55:13,24] [warn] Local [6d31935b]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:55:13,24] [warn] Local [6d31935b]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:55:13,24] [warn] Local [6d31935b]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:55:13,24] [warn] Local [6d31935b]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:55:14,34] [info] WorkflowExecutionActor-6d31935b-042f-4566-b5b7-a155b681d22c [6d31935b]: Starting theiacov_fasta.version_capture [2024-01-10 13:55:16,07] [info] Not triggering log of restart checking token queue status. Effective log interval = None [2024-01-10 13:55:16,11] [info] Not triggering log of execution token queue status. Effective log interval = None [2024-01-10 13:55:17,48] [info] WorkflowExecutionActor-6d31935b-042f-4566-b5b7-a155b681d22c [6d31935b]: Starting theiacov_fasta.abricate_flu, theiacov_fasta.pangolin4 [2024-01-10 13:55:21,12] [info] Assigned new job execution tokens to the following groups: 6d31935b: 3 [2024-01-10 13:55:22,17] [warn] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.version_capture:NA:1]: Unrecognized runtime attribute keys: dx_instance_type, disks, cpu, memory [2024-01-10 13:55:22,23] [warn] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.pangolin4:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 13:55:22,25] [warn] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.abricate_flu:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 13:55:22,27] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.version_capture:NA:1]: PHB_Version="PHB v1.2.1" date +"%Y-%m-%d" > TODAY echo "$PHB_Version" > PHB_VERSION [2024-01-10 13:55:22,27] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.pangolin4:NA:1]: set -e # date and version capture date | tee DATE { pangolin --all-versions && usher --version; } | tr '\n' ';' | cut -f -6 -d ';' | tee VERSION_PANGOLIN_ALL pangolin "/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-pangolin4/inputs/1305340193/clearlabs.fasta.gz" \ \ --min-length 10000 \ --max-ambig 0.5 \ --expanded-lineage \ \ \ --outfile "fasta.pangolin_report.csv" \ --verbose \ python3 < fasta_abricate_hits.tsv # capturing flu type (A or B based on M1 hit) and subtype (e.g. H1 and N1 based on HA/NA hits) ## awk for gene column ($6) to grab subtype ($15) cat fasta_abricate_hits.tsv | awk -F '\t' '{if ($6=="M1") print $15}' > FLU_TYPE HA_hit=$(cat fasta_abricate_hits.tsv | awk -F '\t' '{if ($6=="HA") print $15 }') NA_hit=$(cat fasta_abricate_hits.tsv | awk -F '\t' '{if ($6=="NA") print $15 }') if [[ ! (-z "${HA_hit}") && ! (-z "${NA_hit}") ]]; then flu_subtype="${HA_hit}${NA_hit}" && echo "$flu_subtype" > FLU_SUBTYPE fi if [[ -z "${HA_hit}" ]]; then flu_subtype="${NA_hit}" && echo "$flu_subtype" > FLU_SUBTYPE elif [[ -z "${NA_hit}" ]]; then flu_subtype="${HA_hit}" && echo "$flu_subtype" > FLU_SUBTYPE else flu_subtype="${HA_hit}${NA_hit}" && echo "$flu_subtype" > FLU_SUBTYPE fi #flu_subtype="${HA_hit}${NA_hit}" && echo "$flu_subtype" > FLU_SUBTYPE # set nextclade variables based on subptype run_nextclade=true touch NEXTCLADE_REF_HA NEXTCLADE_REF_NA NEXTCLADE_NAME_HA NEXTCLADE_NAME_NA NEXTCLADE_DS_TAG_HA NEXTCLADE_DS_TAG_NA if [ "${flu_subtype}" == "H1N1" ]; then echo "flu_h1n1pdm_ha" > NEXTCLADE_NAME_HA echo "MW626062" > NEXTCLADE_REF_HA echo "" > NEXTCLADE_DS_TAG_HA echo "flu_h1n1pdm_na" > NEXTCLADE_NAME_NA echo "MW626056" > NEXTCLADE_REF_NA echo "" > NEXTCLADE_DS_TAG_NA elif [ "${flu_subtype}" == "H3N2" ]; then echo "flu_h3n2_ha" > NEXTCLADE_NAME_HA echo "CY163680" > NEXTCLADE_REF_HA echo "" > NEXTCLADE_DS_TAG_HA echo "flu_h3n2_na" > NEXTCLADE_NAME_NA echo "EPI1857215" > NEXTCLADE_REF_NA echo "" > NEXTCLADE_DS_TAG_NA elif [ "${flu_subtype}" == "Victoria" ]; then echo "flu_vic_ha" > NEXTCLADE_NAME_HA echo "KX058884" > NEXTCLADE_REF_HA echo "" > NEXTCLADE_DS_TAG_HA echo "flu_vic_na" > NEXTCLADE_NAME_NA echo "CY073894" > NEXTCLADE_REF_NA echo "" > NEXTCLADE_DS_TAG_NA elif [ "${flu_subtype}" == "Yamagata" ]; then echo "flu_yam_ha" > NEXTCLADE_NAME_HA echo "JN993010" > NEXTCLADE_REF_HA echo "" > NEXTCLADE_DS_TAG_HA # this makes no biological sense, but avoids errors with nextclade echo "flu_vic_na" > NEXTCLADE_NAME_NA echo "CY073894" > NEXTCLADE_REF_NA echo "" > NEXTCLADE_DS_TAG_NA else run_nextclade=false fi echo ${run_nextclade} > RUN_NEXTCLADE [2024-01-10 13:55:22,41] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.version_capture:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-version_capture/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-version_capture/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-version_capture:/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-version_capture:delegated \ us-docker.pkg.dev/general-theiagen/ubuntu/ubuntu@sha256:9a35cef02b57290a7c919824b632897fbe6db3aa18c930110b04d81fcad458ff /cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-version_capture/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-version_capture/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-version_capture/execution/docker_cid # return exit code exit $rc [2024-01-10 13:55:22,41] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.pangolin4:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-pangolin4/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-pangolin4/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-pangolin4:/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-pangolin4:delegated \ us-docker.pkg.dev/general-theiagen/staphb/pangolin@sha256:54b03e235ee04f007417cb934d1a263ce73bbd16aa82134efa680bd5d856c38d /cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-pangolin4/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-pangolin4/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-pangolin4/execution/docker_cid # return exit code exit $rc [2024-01-10 13:55:22,42] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.abricate_flu:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-abricate_flu/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-abricate_flu/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-abricate_flu:/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-abricate_flu:delegated \ us-docker.pkg.dev/general-theiagen/staphb/abricate@sha256:8a5a7ef82ed6d1f8bddd97197e93a55efc467e3ba978830126c97919ca60855d /cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-abricate_flu/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-abricate_flu/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-abricate_flu/execution/docker_cid # return exit code exit $rc [2024-01-10 13:55:25,98] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.pangolin4:NA:1]: job id: 42037 [2024-01-10 13:55:25,98] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.abricate_flu:NA:1]: job id: 42038 [2024-01-10 13:55:25,98] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.version_capture:NA:1]: job id: 42035 [2024-01-10 13:55:25,98] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.pangolin4:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 13:55:25,98] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.abricate_flu:NA:1]: Status change from - to Done [2024-01-10 13:55:25,99] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.version_capture:NA:1]: Status change from - to Done [2024-01-10 13:55:35,87] [info] b1079133-79da-47ea-87f4-4d12c22a7fef-SubWorkflowActor-SubWorkflow-organism_parameters:-1:1 [b1079133]: Workflow organism_parameters complete. Final Outputs: { "organism_parameters.vadr_maxlen": 30000, "organism_parameters.primer_bed": "gs://theiagen-public-files/terra/theiacov-files/empty.bed", "organism_parameters.nextclade_dataset_name": "sars-cov-2", "organism_parameters.kraken_target_organism": "", "organism_parameters.genome_len": 29903, "organism_parameters.vadr_opts": "--noseqnamemax --glsearch -s -r --nomisc --mkey sarscov2 --lowsim5seq 6 --lowsim3seq 6 --alt_fail lowscore,insertnn,deletinn --out_allfasta", "organism_parameters.nextclade_dataset_tag": "2023-08-17T12:00:00Z", "organism_parameters.reference": "tests/inputs/completely-empty-for-test.txt", "organism_parameters.nextclade_dataset_reference": "MN908947", "organism_parameters.reference_gff": "gs://theiagen-public-files/terra/theiacov-files/empty.gff3" } [2024-01-10 13:55:37,86] [info] WorkflowExecutionActor-6d31935b-042f-4566-b5b7-a155b681d22c [6d31935b]: Starting theiacov_fasta.consensus_qc [2024-01-10 13:55:39,90] [info] WorkflowExecutionActor-6d31935b-042f-4566-b5b7-a155b681d22c [6d31935b]: Starting theiacov_fasta.nextclade [2024-01-10 13:55:41,11] [info] Assigned new job execution tokens to the following groups: 6d31935b: 2 [2024-01-10 13:55:41,33] [warn] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.nextclade:NA:1]: Unrecognized runtime attribute keys: disk, dx_instance_type, disks, cpu, memory [2024-01-10 13:55:41,34] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.nextclade:NA:1]: NEXTCLADE_VERSION="$(nextclade --version)" echo $NEXTCLADE_VERSION > NEXTCLADE_VERSION nextclade dataset get --name="sars-cov-2" --reference="MN908947" --tag="2023-08-17T12:00:00Z" -o nextclade_dataset_dir --verbose set -e nextclade run \ --input-dataset=nextclade_dataset_dir/ \ \ \ \ \ \ \ --output-json "clearlabs.fasta.gz".nextclade.json \ --output-tsv "clearlabs.fasta.gz".nextclade.tsv \ --output-tree "clearlabs.fasta.gz".nextclade.auspice.json \ --output-all=. \ "/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-nextclade/inputs/1305340193/clearlabs.fasta.gz" [2024-01-10 13:55:41,35] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.nextclade:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-nextclade/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-nextclade/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-nextclade:/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-nextclade:delegated \ us-docker.pkg.dev/general-theiagen/nextstrain/nextclade@sha256:f4440021c7f854433bd56433024f38199de27e0ef657e11d5fb28aad4f265a48 /cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-nextclade/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-nextclade/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-nextclade/execution/docker_cid # return exit code exit $rc [2024-01-10 13:55:41,38] [warn] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.consensus_qc:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 13:55:41,39] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.consensus_qc:NA:1]: if [ -s "/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-consensus_qc/inputs/-273586876/completely-empty-for-test.txt" ] ; then GENOME_LEN=$(grep -v ">" /cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-consensus_qc/inputs/-273586876/completely-empty-for-test.txt | tr --delete '\n' | wc -c) elif [ 29903 ] ; then GENOME_LEN=29903 else # set SC2 default GENOME_LEN=29903 fi # capture date and version date | tee DATE num_N=$( grep -v ">" /cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-consensus_qc/inputs/1305340193/clearlabs.fasta.gz | grep -o 'N' | wc -l ) if [ -z "$num_N" ] ; then num_N="0" ; fi echo $num_N | tee NUM_N num_ACTG=$( grep -v ">" /cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-consensus_qc/inputs/1305340193/clearlabs.fasta.gz | grep -o -E "C|A|T|G" | wc -l ) if [ -z "$num_ACTG" ] ; then num_ACTG="0" ; fi echo $num_ACTG | tee NUM_ACTG # calculate percent coverage (Wu Han-1 genome length: 29903bp) python3 -c "print ( round( ($num_ACTG / $GENOME_LEN ) * 100, 2 ) )" | tee PERCENT_REF_COVERAGE num_degenerate=$( grep -v ">" /cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-consensus_qc/inputs/1305340193/clearlabs.fasta.gz | grep -o -E "B|D|E|F|H|I|J|K|L|M|O|P|Q|R|S|U|V|W|X|Y|Z" | wc -l ) if [ -z "$num_degenerate" ] ; then num_degenerate="0" ; fi echo $num_degenerate | tee NUM_DEGENERATE num_total=$( grep -v ">" /cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-consensus_qc/inputs/1305340193/clearlabs.fasta.gz | grep -o -E '[A-Z]' | wc -l ) if [ -z "$num_total" ] ; then num_total="0" ; fi echo $num_total | tee NUM_TOTAL [2024-01-10 13:55:41,43] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.consensus_qc:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-consensus_qc/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-consensus_qc/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-consensus_qc:/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-consensus_qc:delegated \ us-docker.pkg.dev/general-theiagen/theiagen/utility@sha256:b40130a48b94d64db4520ad8580c72cbcb8ea16bba486bc315096c2c3ec8cace /cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-consensus_qc/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-consensus_qc/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-consensus_qc/execution/docker_cid # return exit code exit $rc [2024-01-10 13:55:45,96] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.nextclade:NA:1]: job id: 42765 [2024-01-10 13:55:45,96] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.consensus_qc:NA:1]: job id: 42778 [2024-01-10 13:55:45,96] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.nextclade:NA:1]: Status change from - to Done [2024-01-10 13:55:45,96] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.consensus_qc:NA:1]: Status change from - to Done [2024-01-10 13:55:47,04] [info] WorkflowExecutionActor-6d31935b-042f-4566-b5b7-a155b681d22c [6d31935b]: Starting theiacov_fasta.vadr [2024-01-10 13:55:49,50] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.pangolin4:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 13:55:51,11] [info] Assigned new job execution tokens to the following groups: 6d31935b: 1 [2024-01-10 13:55:51,12] [info] WorkflowExecutionActor-6d31935b-042f-4566-b5b7-a155b681d22c [6d31935b]: Starting theiacov_fasta.nextclade_output_parser [2024-01-10 13:55:51,45] [warn] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.vadr:NA:1]: Unrecognized runtime attribute keys: dx_instance_type, cpu, memory [2024-01-10 13:55:51,46] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.vadr:NA:1]: set -e if [ 0 -gt 10000 ]; then # remove terminal ambiguous nucleotides /opt/vadr/vadr/miniscripts/fasta-trim-terminal-ambigs.pl \ "/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-vadr/inputs/1305340193/clearlabs.fasta.gz" \ --minlen 50 \ --maxlen 30000 \ > "clearlabs.fasta.gz_trimmed.fasta" # run VADR # --split and --cpu must be used in conjuction v-annotate.pl \ --split --cpu 2 \ --noseqnamemax --glsearch -s -r --nomisc --mkey sarscov2 --lowsim5seq 6 --lowsim3seq 6 --alt_fail lowscore,insertnn,deletinn --out_allfasta \ "clearlabs.fasta.gz_trimmed.fasta" \ "clearlabs.fasta.gz" # package everything for output tar -C "clearlabs.fasta.gz" -czvf "clearlabs.fasta.gz.vadr.tar.gz" . # package up FASTA files into zip file for output. Note: this will work whether the --out_allfasta flag is included or not (there are just more when the option is used) mkdir -v vadr_fasta_files cp -v clearlabs.fasta.gz/*.fa vadr_fasta_files zip clearlabs.fasta.gz_vadr-fasta-files.zip vadr_fasta_files/*.fa # prep alerts into a tsv file for parsing cut -f 5 "clearlabs.fasta.gz/clearlabs.fasta.gz.vadr.alt.list" | tail -n +2 > "clearlabs.fasta.gz.vadr.alerts.tsv" cat "clearlabs.fasta.gz.vadr.alerts.tsv" | wc -l > NUM_ALERTS else echo "VADR skipped due to poor assembly; assembly length (unambiguous) = 0" > NUM_ALERTS fi [2024-01-10 13:55:51,47] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.vadr:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-vadr/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-vadr/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-vadr:/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-vadr:delegated \ us-docker.pkg.dev/general-theiagen/staphb/vadr@sha256:f15a73a6f6e0802c3a9c05e7c602740e7029a2e1ccc14aa6935b459be87c98d7 /cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-vadr/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-vadr/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-vadr/execution/docker_cid # return exit code exit $rc [2024-01-10 13:55:55,94] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.vadr:NA:1]: job id: 43234 [2024-01-10 13:55:55,95] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.vadr:NA:1]: Status change from - to Done [2024-01-10 13:56:01,11] [info] Assigned new job execution tokens to the following groups: 6d31935b: 1 [2024-01-10 13:56:01,45] [warn] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.nextclade_output_parser:NA:1]: Unrecognized runtime attribute keys: disk, dx_instance_type, disks, cpu, memory [2024-01-10 13:56:01,45] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-nextclade_output_parser/inputs/-617491802/clearlabs.fasta.gz.nextclade.tsv -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-nextclade/execution/clearlabs.fasta.gz.nextclade.tsv: Operation not permitted [2024-01-10 13:56:01,46] [info] BackgroundConfigAsyncJobExecutionActor [6d31935btheiacov_fasta.nextclade_output_parser:NA:1]: # Set WDL input variable to input.tsv file cat "/cromwell-executions/theiacov_fasta/6d31935b-042f-4566-b5b7-a155b681d22c/call-nextclade_output_parser/inputs/-617491802/clearlabs.fasta.gz.nextclade.tsv" > input.tsv touch TAMIFLU_AASUBS # Parse outputs using python3 python3 <

✅ theiacov_fasta via cromwell. exit code was 0

kapsakcj commented 9 months ago
TheiaCov_ClearLabs test via `cromwell` : ```bash $ cromwell run -i ./tests/inputs/theiacov/wf_theiacov_clearlabs.json -m metadata.json ./workflows/theiacov/wf_theiacov_clearlabs.wdl [2024-01-10 13:58:46,31] [info] Running with database db.url = jdbc:hsqldb:mem:cec08c26-22cd-43c2-97f2-0fdcaad5f990;shutdown=false;hsqldb.tx=mvcc Jan 10, 2024 1:58:46 PM liquibase.servicelocator INFO: Cannot load service: liquibase.license.LicenseService: liquibase.license.pro.DaticalTrueLicenseService Unable to get public no-arg constructor Jan 10, 2024 1:58:46 PM liquibase.database INFO: Set default schema name to PUBLIC Jan 10, 2024 1:58:47 PM liquibase.lockservice INFO: Successfully acquired change log lock Jan 10, 2024 1:58:48 PM liquibase.changelog INFO: Creating database history table with name: PUBLIC.DATABASECHANGELOG Jan 10, 2024 1:58:48 PM liquibase.changelog INFO: Reading from PUBLIC.DATABASECHANGELOG Running Changeset: changesets/db_schema.xml::db_schema_other_table_alldb::scottfrazer Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table WORKFLOW_EXECUTION created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table EXECUTION created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table JES_JOB created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table LOCAL_JOB created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/db_schema.xml::db_schema_other_table_alldb::scottfrazer ran successfully in 21ms Running Changeset: changesets/db_schema.xml::db_schema_symbol_table_otherdb::scottfrazer Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table SYMBOL created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/db_schema.xml::db_schema_symbol_table_otherdb::scottfrazer ran successfully in 6ms Running Changeset: changesets/db_schema.xml::db_schema_constraints::scottfrazer Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index STATUS_IDX created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to SYMBOL (WORKFLOW_EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION (WORKFLOW_EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to JES_JOB (EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to LOCAL_JOB (EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/db_schema.xml::db_schema_constraints::scottfrazer ran successfully in 19ms Running Changeset: changesets/symbol_iteration_null.xml::symbol_iteration_null::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Null constraint dropped from SYMBOL.ITERATION Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/symbol_iteration_null.xml::symbol_iteration_null::kshakir ran successfully in 2ms Running Changeset: changesets/wdl_and_inputs.xml::WORKFLOW_EXECUTION_AUX::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table WORKFLOW_EXECUTION_AUX created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/wdl_and_inputs.xml::WORKFLOW_EXECUTION_AUX::mcovarr ran successfully in 3ms Running Changeset: changesets/wdl_and_inputs.xml::WE_AUX_CONSTRAINTS::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to WORKFLOW_EXECUTION_AUX (WORKFLOW_EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_EXECUTION_AUX(WORKFLOW_EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/wdl_and_inputs.xml::WE_AUX_CONSTRAINTS::mcovarr ran successfully in 5ms Running Changeset: changesets/drop_workflow_uri_and_local_command.xml::drop_workflow_uri_and_local_command::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column WORKFLOW_EXECUTION.WDL_URI dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column LOCAL_JOB.COMMAND dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/drop_workflow_uri_and_local_command.xml::drop_workflow_uri_and_local_command::kshakir ran successfully in 6ms Running Changeset: changesets/local_job_allow_null.xml::local_job_allow_null::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Null constraint dropped from LOCAL_JOB.PID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Null constraint dropped from LOCAL_JOB.RC Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/local_job_allow_null.xml::local_job_allow_null::kshakir ran successfully in 4ms Running Changeset: changesets/symbol_iteration_not_null.xml::symbol_iteration_not_null::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: NOT NULL constraint has been added to SYMBOL.ITERATION Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/symbol_iteration_not_null.xml::symbol_iteration_not_null::kshakir ran successfully in 4ms Running Changeset: changesets/add_unique_constraints.xml::add_unique_constraints::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_EXECUTION(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to EXECUTION(WORKFLOW_EXECUTION_ID, CALL_FQN) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to SYMBOL(WORKFLOW_EXECUTION_ID, SCOPE, NAME, ITERATION, IO) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to LOCAL_JOB(EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to JES_JOB(EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/add_unique_constraints.xml::add_unique_constraints::kshakir ran successfully in 11ms Running Changeset: changesets/lengthen_wdl_value.xml::LENGTHEN_WDL_VALUE::sfrazer Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: SYMBOL.WDL_VALUE datatype was changed to LONGVARCHAR Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/lengthen_wdl_value.xml::LENGTHEN_WDL_VALUE::sfrazer ran successfully in 3ms Running Changeset: changesets/add_index_in_execution.xml::add-index-in-execution::tjeandet Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Columns INDEX(INT) added to EXECUTION Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: NOT NULL constraint has been added to EXECUTION.INDEX Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/add_index_in_execution.xml::add-index-in-execution::tjeandet ran successfully in 5ms Running Changeset: changesets/rename_iteration_to_index.xml::rename-iteration-to-index::tjeandet Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column SYMBOL.ITERATION renamed to INDEX Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/rename_iteration_to_index.xml::rename-iteration-to-index::tjeandet ran successfully in 2ms Running Changeset: changesets/sge.xml::sge::scottfrazer Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table SGE_JOB created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/sge.xml::sge::scottfrazer ran successfully in 2ms Running Changeset: changesets/sge.xml::db_schema_constraints::scottfrazer Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to SGE_JOB (EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/sge.xml::db_schema_constraints::scottfrazer ran successfully in 2ms Running Changeset: changesets/change_execution_unique_constraint.xml::change_execution_unique_constraint::tjeandet Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key FK_EXECUTION_WORKFLOW_EXECUTION_ID dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint UK_EX_WORKFLOW_EXECUTION_ID dropped from EXECUTION Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to EXECUTION(WORKFLOW_EXECUTION_ID, CALL_FQN, INDEX) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION (WORKFLOW_EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column EXECUTION.INDEX renamed to IDX Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/change_execution_unique_constraint.xml::change_execution_unique_constraint::tjeandet ran successfully in 10ms Running Changeset: changesets/rc.xml::rc::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column LOCAL_JOB.RC dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Columns RC(INT) added to EXECUTION Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/rc.xml::rc::mcovarr ran successfully in 4ms Running Changeset: changesets/workflow_options.xml::workflow-options::sfrazer Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Columns WORKFLOW_OPTIONS(LONGVARCHAR) added to WORKFLOW_EXECUTION_AUX Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_options.xml::workflow-options::sfrazer ran successfully in 3ms Running Changeset: changesets/jes_id_update.xml::jes_id_update::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: JES_JOB.JES_ID datatype was changed to VARCHAR(100) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: JES_JOB.JES_STATUS datatype was changed to VARCHAR(100) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/jes_id_update.xml::jes_id_update::chrisl ran successfully in 5ms Running Changeset: changesets/optional_sge_ids.xml::optional_sge_ids::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: SGE_JOB.SGE_JOB_NUMBER datatype was changed to INT Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/optional_sge_ids.xml::optional_sge_ids::chrisl ran successfully in 2ms Running Changeset: changesets/add_start_end_time_in_execution.xml::add-start-end-time-in-execution::jgentry Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Columns START_DT(datetime) added to EXECUTION Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Columns END_DT(datetime) added to EXECUTION Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/add_start_end_time_in_execution.xml::add-start-end-time-in-execution::jgentry ran successfully in 5ms Running Changeset: changesets/rename_jes_id.xml::rename-jes-id-to-jes-run-id::sfrazer Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column JES_JOB.JES_ID renamed to JES_RUN_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/rename_jes_id.xml::rename-jes-id-to-jes-run-id::sfrazer ran successfully in 2ms Running Changeset: changesets/add_workflow_name.xml::workflow-name::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Columns WORKFLOW_NAME(VARCHAR(500)) added to WORKFLOW_EXECUTION Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/add_workflow_name.xml::workflow-name::mcovarr ran successfully in 4ms Running Changeset: changesets/top_level_output.xml::top_level_output::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Columns REPORTABLE_RESULT(TINYINT) added to SYMBOL Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/top_level_output.xml::top_level_output::chrisl ran successfully in 3ms Running Changeset: changesets/workflow_execution_aux_not_null.xml::workflow_execution_aux_not_null::tjeandet Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_EXECUTION_AUX.WORKFLOW_EXECUTION_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_EXECUTION_AUX.WDL_SOURCE Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_EXECUTION_AUX.JSON_INPUTS Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_EXECUTION_AUX.WORKFLOW_OPTIONS Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_execution_aux_not_null.xml::workflow_execution_aux_not_null::tjeandet ran successfully in 9ms Running Changeset: changesets/call_result_caching.xml::call_result_caching::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Columns ALLOWS_RESULT_REUSE(BOOLEAN),DOCKER_IMAGE_HASH(VARCHAR(100)),RESULTS_CLONED_FROM(INT),EXECUTION_HASH(VARCHAR(100)) added to EXECUTION Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index HASH_INDEX created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Columns HASH(VARCHAR(100)) added to SYMBOL Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION (RESULTS_CLONED_FROM) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/call_result_caching.xml::call_result_caching::chrisl ran successfully in 18ms Running Changeset: changesets/events_table.xml::execution_event_table::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table EXECUTION_EVENT created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION_EVENT (EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to EXECUTION_EVENT(EXECUTION_ID, DESCRIPTION) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/events_table.xml::execution_event_table::chrisl ran successfully in 9ms Running Changeset: changesets/sync_not_null_constraints.xml::job_not_null_columns::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Null constraint dropped from JES_JOB.JES_RUN_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Null constraint dropped from JES_JOB.JES_STATUS Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Null constraint dropped from SGE_JOB.SGE_JOB_NUMBER Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/sync_not_null_constraints.xml::job_not_null_columns::kshakir ran successfully in 5ms Running Changeset: changesets/sync_not_null_constraints.xml::events_end_dt_not_null::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: NOT NULL constraint has been added to EXECUTION_EVENT.END_DT Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/sync_not_null_constraints.xml::events_end_dt_not_null::kshakir ran successfully in 2ms Running Changeset: changesets/sge_job_execution_unique_key.xml::sge_job_execution_unique_key::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to SGE_JOB(EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/sge_job_execution_unique_key.xml::sge_job_execution_unique_key::kshakir ran successfully in 3ms Running Changeset: changesets/add_attempt_in_execution.xml::add-attempt-in-execution::tjeandet Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Columns ATTEMPT(INT) added to EXECUTION Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: NOT NULL constraint has been added to EXECUTION.ATTEMPT Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key FK_EXECUTION_WORKFLOW dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint UK_WORKFLOW_CALL_INDEX dropped from EXECUTION Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to EXECUTION(WORKFLOW_EXECUTION_ID, CALL_FQN, IDX, ATTEMPT) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION (WORKFLOW_EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/add_attempt_in_execution.xml::add-attempt-in-execution::tjeandet ran successfully in 19ms Running Changeset: changesets/execution_backend_info.xml::CREATE_EXECUTION_INFO::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table EXECUTION_INFO created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::CREATE_EXECUTION_INFO::mcovarr ran successfully in 4ms Running Changeset: changesets/execution_backend_info.xml::add_foreign_key_constraints::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION_INFO (EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::add_foreign_key_constraints::mcovarr ran successfully in 3ms Running Changeset: changesets/execution_backend_info.xml::add_backend_type_column::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Columns BACKEND_TYPE(VARCHAR(255)) added to EXECUTION Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::add_backend_type_column::mcovarr ran successfully in 3ms Running Changeset: changesets/execution_backend_info.xml::insert_data_from_backend_tables::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::insert_data_from_backend_tables::mcovarr ran successfully in 12ms Running Changeset: changesets/execution_backend_info.xml::set_backend_type::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::set_backend_type::mcovarr ran successfully in 9ms Running Changeset: changesets/execution_backend_info.xml::drop_job_tables::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table JES_JOB dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table SGE_JOB dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table LOCAL_JOB dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::drop_job_tables::mcovarr ran successfully in 9ms Running Changeset: changesets/execution_backend_info.xml::execution_info_uniqueness_constraint::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to EXECUTION_INFO(EXECUTION_ID, INFO_KEY) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::execution_info_uniqueness_constraint::mcovarr ran successfully in 4ms Running Changeset: changesets/execution_backend_info.xml::execution_backend_type_not_null::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: NOT NULL constraint has been added to EXECUTION.BACKEND_TYPE Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::execution_backend_type_not_null::mcovarr ran successfully in 3ms Running Changeset: changesets/runtime_attributes_table.xml::runtime-attributes-table::tjeandet Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table RUNTIME_ATTRIBUTES created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to RUNTIME_ATTRIBUTES (EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to RUNTIME_ATTRIBUTES(EXECUTION_ID, ATTRIBUTE_NAME) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/runtime_attributes_table.xml::runtime-attributes-table::tjeandet ran successfully in 8ms Running Changeset: changesets/failure_table.xml::failure_table::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table FAILURE_EVENT created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to FAILURE_EVENT (EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to FAILURE_EVENT (WORKFLOW_EXECUTION_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/failure_table.xml::failure_table::chrisl ran successfully in 9ms Running Changeset: changesets/metadata_journal.xml::metadata_journal::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table METADATA_JOURNAL created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal.xml::metadata_journal::mcovarr ran successfully in 3ms Running Changeset: changesets/metadata_journal.xml::metadata_workflow_index::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index METADATA_WORKFLOW_IDX created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal.xml::metadata_workflow_index::mcovarr ran successfully in 2ms Running Changeset: changesets/metadata_journal.xml::metadata_job_index::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index METADATA_JOB_IDX created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal.xml::metadata_job_index::mcovarr ran successfully in 2ms Running Changeset: changesets/metadata_journal.xml::metadata_job_and_key_index::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index METADATA_JOB_AND_KEY_IDX created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal.xml::metadata_job_and_key_index::mcovarr ran successfully in 2ms Running Changeset: changesets/metadata_typed_values.xml::metadata_typed_values::tjeandet Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Columns METADATA_VALUE_TYPE(VARCHAR(10)) added to METADATA_JOURNAL Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/metadata_typed_values.xml::metadata_typed_values::tjeandet ran successfully in 3ms Running Changeset: changesets/workflow_metadata_summary.xml::metadata_journal_id_int_to_big_int::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: METADATA_JOURNAL.METADATA_JOURNAL_ID datatype was changed to BIGINT Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Auto-increment added to METADATA_JOURNAL.METADATA_JOURNAL_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::metadata_journal_id_int_to_big_int::kshakir ran successfully in 4ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_summary::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_summary::mcovarr ran successfully in 4ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_uuid_idx::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_METADATA_SUMMARY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_uuid_idx::mcovarr ran successfully in 2ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_name_idx::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_NAME_IDX created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_name_idx::mcovarr ran successfully in 2ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_status_idx::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_STATUS_IDX created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_status_idx::mcovarr ran successfully in 2ms Running Changeset: changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_subsecond_timestamp::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: METADATA_JOURNAL.METADATA_TIMESTAMP datatype was changed to DATETIME(6) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_subsecond_timestamp::mcovarr ran successfully in 2ms Running Changeset: changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_timestamp_not_null::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: NOT NULL constraint has been added to METADATA_JOURNAL.METADATA_TIMESTAMP Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_timestamp_not_null::mcovarr ran successfully in 2ms Running Changeset: changesets/workflow_store.xml::WORKFLOW_STORE::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table WORKFLOW_STORE created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store.xml::WORKFLOW_STORE::chrisl ran successfully in 4ms Running Changeset: changesets/workflow_store.xml::workflow_store_uuid_index::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_STORE(WORKFLOW_UUID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store.xml::workflow_store_uuid_index::kshakir ran successfully in 3ms Running Changeset: changesets/workflow_store.xml::workflow_store_state_index::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index WORKFLOW_STORE_STATE_IDX created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store.xml::workflow_store_state_index::chrisl ran successfully in 2ms Running Changeset: changesets/backend_KV_Store.xml::BACKEND_KV_STORE::rmunshi Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table BACKEND_KV_STORE created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/backend_KV_Store.xml::BACKEND_KV_STORE::rmunshi ran successfully in 3ms Running Changeset: changesets/backend_KV_Store.xml::backend_KV_store_job_key_constraint::rmunshi Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to BACKEND_KV_STORE(WORKFLOW_EXECUTION_UUID, CALL_FQN, JOB_SCATTER_INDEX, JOB_RETRY_ATTEMPT, STORE_KEY) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/backend_KV_Store.xml::backend_KV_store_job_key_constraint::rmunshi ran successfully in 3ms Running Changeset: changesets/job_store.xml::JOB_STORE::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table JOB_STORE created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/job_store.xml::JOB_STORE::chrisl ran successfully in 4ms Running Changeset: changesets/job_store.xml::job_store_uuid_index::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index JOB_STORE_UUID_IDX created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/job_store.xml::job_store_uuid_index::chrisl ran successfully in 2ms Running Changeset: changesets/job_store.xml::job_store_jobkey_index::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE(WORKFLOW_UUID, CALL_FQN, JOB_SCATTER_INDEX, JOB_RETRY_ATTEMPT) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/job_store.xml::job_store_jobkey_index::chrisl ran successfully in 4ms Running Changeset: changesets/callcaching.xml::call_caching_result_metainfo::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_METAINFO created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_metainfo::chrisl ran successfully in 4ms Running Changeset: changesets/callcaching.xml::call_caching_result_metainfo_uniqueness::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_RESULT_METAINFO(WORKFLOW_UUID, CALL_FQN, JOB_SCATTER_INDEX) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_metainfo_uniqueness::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_hashes::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table CALL_CACHING_HASH created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_hashes::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_hashes_uniqueness::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_HASH(HASH_KEY, RESULT_METAINFO_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_hashes_uniqueness::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_hashes_uniqueness_foreign_key::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH (RESULT_METAINFO_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_hashes_uniqueness_foreign_key::chrisl ran successfully in 2ms Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_SIMPLETON created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_simpletons::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons_uniqueness::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_RESULT_SIMPLETON(SIMPLETON_KEY, RESULT_METAINFO_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_simpletons_uniqueness::chrisl ran successfully in 2ms Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons_foreign_key::chrisl Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_RESULT_SIMPLETON (RESULT_METAINFO_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_simpletons_foreign_key::chrisl ran successfully in 2ms Running Changeset: changesets/call_caching_allow_result_reuse_fix.xml::call_caching_fix_boolean::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: CALL_CACHING_RESULT_METAINFO.ALLOW_RESULT_REUSE datatype was changed to BOOLEAN Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_allow_result_reuse_fix.xml::call_caching_fix_boolean::mcovarr ran successfully in 2ms Running Changeset: changesets/call_caching_allow_result_reuse_fix.xml::call_caching_reuse_default::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Default value added to CALL_CACHING_RESULT_METAINFO.ALLOW_RESULT_REUSE Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_allow_result_reuse_fix.xml::call_caching_reuse_default::mcovarr ran successfully in 2ms Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table JOB_STORE_RESULT_SIMPLETON created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_result_simpletons::mcovarr ran successfully in 3ms Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons_uniqueness::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE_RESULT_SIMPLETON(SIMPLETON_KEY, JOB_STORE_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_result_simpletons_uniqueness::mcovarr ran successfully in 2ms Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons_foreign_key::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_RESULT_SIMPLETON (JOB_STORE_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_result_simpletons_foreign_key::mcovarr ran successfully in 2ms Running Changeset: changesets/job_store_simpletons.xml::job_store_remove_job_output::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column JOB_STORE.JOB_OUTPUT dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_remove_job_output::mcovarr ran successfully in 2ms Running Changeset: changesets/restart_and_recover_migration.xml::WORKFLOW_STORE_MIGRATION::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::WORKFLOW_STORE_MIGRATION::mcovarr ran successfully in 5ms Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_MIGRATION::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::JOB_STORE_MIGRATION::mcovarr ran successfully in 8ms Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_VALUE_NULLABLE::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Null constraint dropped from JOB_STORE_RESULT_SIMPLETON.SIMPLETON_VALUE Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_VALUE_NULLABLE::mcovarr ran successfully in 18ms Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_MIGRATION::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: WORKFLOW_EXECUTION + EXECUTION + SYMBOL + JOB_STORE -> JOB_STORE_RESULT_SIMPLETON migration complete. Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_MIGRATION::mcovarr ran successfully in 2ms Running Changeset: changesets/restart_and_recover_migration.xml::BACKEND_KV_STORE_MIGRATION::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::BACKEND_KV_STORE_MIGRATION::mcovarr ran successfully in 3ms Running Changeset: changesets/restart_and_recover_migration.xml::WORKFLOW_OPTIONS_RENAME_MIGRATION::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Workflow option renaming migration complete. Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::WORKFLOW_OPTIONS_RENAME_MIGRATION::mcovarr ran successfully in 2ms Running Changeset: changesets/summary_status_table.xml::summary_status_table::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table SUMMARY_STATUS created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/summary_status_table.xml::summary_status_table::kshakir ran successfully in 3ms Running Changeset: changesets/summary_status_table.xml::summary_status_summarized_table_name_index::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS(SUMMARY_TABLE_NAME, SUMMARIZED_TABLE_NAME) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/summary_status_table.xml::summary_status_summarized_table_name_index::kshakir ran successfully in 2ms Running Changeset: changesets/standardize_column_names.xml::workflow_execution_uuid::rmunshi Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column CALL_CACHING_RESULT_METAINFO.WORKFLOW_UUID renamed to WORKFLOW_EXECUTION_UUID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column WORKFLOW_STORE.WORKFLOW_UUID renamed to WORKFLOW_EXECUTION_UUID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column JOB_STORE.WORKFLOW_UUID renamed to WORKFLOW_EXECUTION_UUID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names.xml::workflow_execution_uuid::rmunshi ran successfully in 4ms Running Changeset: changesets/standardize_column_names.xml::call_job_identifiers::rmunshi Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column METADATA_JOURNAL.METADATA_CALL_FQN renamed to CALL_FQN Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column METADATA_JOURNAL.METADATA_CALL_INDEX renamed to JOB_SCATTER_INDEX Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column METADATA_JOURNAL.METADATA_CALL_ATTEMPT renamed to JOB_RETRY_ATTEMPT Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names.xml::call_job_identifiers::rmunshi ran successfully in 4ms Running Changeset: changesets/embiggen_metadata_value.xml::entry_or_journal_existence_xor::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/embiggen_metadata_value.xml::entry_or_journal_existence_xor::mcovarr ran successfully in 15ms Running Changeset: changesets/embiggen_metadata_value.xml::embiggen_metadata_entry::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Marking ChangeSet: changesets/embiggen_metadata_value.xml::embiggen_metadata_entry::mcovarr ran despite precondition failure due to onFail='MARK_RAN': changelog.xml : Table PUBLIC.METADATA_ENTRY does not exist Running Changeset: changesets/embiggen_metadata_value.xml::embiggen_metadata_journal::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: METADATA_JOURNAL.METADATA_VALUE datatype was changed to LONGTEXT Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/embiggen_metadata_value.xml::embiggen_metadata_journal::mcovarr ran successfully in 5ms Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus::rmunshi Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table CALL_CACHING_JOB_DETRITUS created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_job_detritus.xml::call_caching_job_detritus::rmunshi ran successfully in 3ms Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus_key_uniqueness::rmunshi Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_JOB_DETRITUS(JOB_DETRITUS_KEY, RESULT_METAINFO_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_job_detritus.xml::call_caching_job_detritus_key_uniqueness::rmunshi ran successfully in 2ms Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus_foreign_key::rmunshi Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_JOB_DETRITUS (RESULT_METAINFO_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_job_detritus.xml::call_caching_job_detritus_foreign_key::rmunshi ran successfully in 2ms Running Changeset: changesets/job_store_tinyints.xml::job_store_fix_job_successful::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: JOB_STORE.JOB_SUCCESSFUL datatype was changed to BOOLEAN Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/job_store_tinyints.xml::job_store_fix_job_successful::mcovarr ran successfully in 1ms Running Changeset: changesets/job_store_tinyints.xml::job_store_fix_job_retryable_failure::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: JOB_STORE.RETRYABLE_FAILURE datatype was changed to BOOLEAN Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/job_store_tinyints.xml::job_store_fix_job_retryable_failure::mcovarr ran successfully in 1ms Running Changeset: changesets/job_store_tinyints.xml::job_store_not_nullable_job_successful::mcovarr Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: NOT NULL constraint has been added to JOB_STORE.JOB_SUCCESSFUL Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/job_store_tinyints.xml::job_store_not_nullable_job_successful::mcovarr ran successfully in 2ms Running Changeset: changesets/standardize_column_names_again.xml::standardize_column_names_again::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index JOB_STORE_UUID_IDX dropped from table JOB_STORE Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_NAME_IDX dropped from table WORKFLOW_METADATA_SUMMARY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_STATUS_IDX dropped from table WORKFLOW_METADATA_SUMMARY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index WORKFLOW_STORE_STATE_IDX dropped from table WORKFLOW_STORE Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_JOB_DETRITUS_KEY dropped from CALL_CACHING_JOB_DETRITUS Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_RESULT_METAINFO dropped from CALL_CACHING_RESULT_METAINFO Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_HASH dropped from CALL_CACHING_HASH Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_RESULT_SIMPLETON dropped from CALL_CACHING_RESULT_SIMPLETON Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint JOB_STORE_JOBKEY_IDX dropped from JOB_STORE Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint UK_BACKEND_KV_STORE_KEY dropped from BACKEND_KV_STORE Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint UK_JOB_STORE_RESULT_SIMPLETON dropped from JOB_STORE_RESULT_SIMPLETON Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint SUMMARY_STATUS_SUMMARY_TABLE_NAME_SUMMARIZED_TABLE_NAME_INDEX dropped from SUMMARY_STATUS Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint WORKFLOW_METADATA_UUID_IDX dropped from WORKFLOW_METADATA_SUMMARY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint WORKFLOW_STORE_UUID_IDX dropped from WORKFLOW_STORE Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key CCJD_RESULT_METAINFO_ID_FK dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key CCH_RESULT_METAINFO_ID_FK dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key CCRS_RESULT_METAINFO_ID_FK dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key JSRS_JOB_STORE_FK dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table CALL_CACHING_JOB_DETRITUS renamed to CALL_CACHING_DETRITUS_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_METAINFO renamed to CALL_CACHING_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table CALL_CACHING_HASH renamed to CALL_CACHING_HASH_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_SIMPLETON renamed to CALL_CACHING_SIMPLETON_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table BACKEND_KV_STORE renamed to JOB_KEY_VALUE_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table JOB_STORE renamed to JOB_STORE_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table JOB_STORE_RESULT_SIMPLETON renamed to JOB_STORE_SIMPLETON_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table METADATA_JOURNAL renamed to METADATA_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table SUMMARY_STATUS renamed to SUMMARY_STATUS_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY renamed to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table WORKFLOW_STORE renamed to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_JOB_DETRITUS_ID renamed to CALL_CACHING_DETRITUS_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.JOB_DETRITUS_KEY renamed to DETRITUS_KEY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.JOB_DETRITUS_VALUE renamed to DETRITUS_VALUE Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column CALL_CACHING_ENTRY.CALL_CACHING_RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column CALL_CACHING_ENTRY.JOB_SCATTER_INDEX renamed to JOB_INDEX Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column CALL_CACHING_ENTRY.CALL_FQN renamed to CALL_FULLY_QUALIFIED_NAME Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column CALL_CACHING_HASH_ENTRY.RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ID renamed to CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column CALL_CACHING_SIMPLETON_ENTRY.RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_RESULT_SIMPLETON_ID renamed to CALL_CACHING_SIMPLETON_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.JOB_RETRY_ATTEMPT renamed to JOB_ATTEMPT Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.JOB_SCATTER_INDEX renamed to JOB_INDEX Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.BACKEND_KV_STORE_ID renamed to JOB_KEY_VALUE_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.CALL_FQN renamed to CALL_FULLY_QUALIFIED_NAME Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.JOB_RETRY_ATTEMPT renamed to JOB_ATTEMPT Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.JOB_SCATTER_INDEX renamed to JOB_INDEX Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.JOB_STORE_ID renamed to JOB_STORE_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.CALL_FQN renamed to CALL_FULLY_QUALIFIED_NAME Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_ID renamed to JOB_STORE_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_RESULT_SIMPLETON_ID renamed to JOB_STORE_SIMPLETON_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.SUMMARY_STATUS_ID renamed to SUMMARY_STATUS_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column WORKFLOW_METADATA_SUMMARY_ENTRY.END_DT renamed to END_TIMESTAMP Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column WORKFLOW_METADATA_SUMMARY_ENTRY.START_DT renamed to START_TIMESTAMP Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column WORKFLOW_METADATA_SUMMARY_ENTRY.WORKFLOW_METADATA_SUMMARY_ID renamed to WORKFLOW_METADATA_SUMMARY_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Column WORKFLOW_STORE_ENTRY.WORKFLOW_STORE_ID renamed to WORKFLOW_STORE_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_DETRITUS_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_HASH_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_SIMPLETON_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key dropped from JOB_KEY_VALUE_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key dropped from JOB_STORE_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key dropped from JOB_STORE_SIMPLETON_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key dropped from SUMMARY_STATUS_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key dropped from WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key dropped from WORKFLOW_STORE_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_DETRITUS_ENTRY (CALL_CACHING_DETRITUS_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_HASH_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_SIMPLETON_ENTRY (CALL_CACHING_SIMPLETON_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key added to JOB_KEY_VALUE_ENTRY (JOB_KEY_VALUE_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key added to JOB_STORE_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_SIMPLETON_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key added to SUMMARY_STATUS_ENTRY (SUMMARY_STATUS_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key added to WORKFLOW_METADATA_SUMMARY_ENTRY (WORKFLOW_METADATA_SUMMARY_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Primary key added to WORKFLOW_STORE_ENTRY (WORKFLOW_STORE_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_DETRITUS_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_ENTRY.CALL_CACHING_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_SIMPLETON_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Auto-increment added to JOB_KEY_VALUE_ENTRY.JOB_KEY_VALUE_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_ENTRY.JOB_STORE_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_SIMPLETON_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Auto-increment added to SUMMARY_STATUS_ENTRY.SUMMARY_STATUS_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Auto-increment added to WORKFLOW_METADATA_SUMMARY_ENTRY.WORKFLOW_METADATA_SUMMARY_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Auto-increment added to WORKFLOW_STORE_ENTRY.WORKFLOW_STORE_ENTRY_ID Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_DETRITUS_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_SIMPLETON_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_DETRITUS_ENTRY(CALL_CACHING_ENTRY_ID, DETRITUS_KEY) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_HASH_ENTRY(CALL_CACHING_ENTRY_ID, HASH_KEY) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_SIMPLETON_ENTRY(CALL_CACHING_ENTRY_ID, SIMPLETON_KEY) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to JOB_KEY_VALUE_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX, JOB_ATTEMPT, STORE_KEY) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX, JOB_ATTEMPT) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE_SIMPLETON_ENTRY(JOB_STORE_ENTRY_ID, SIMPLETON_KEY) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS_ENTRY(SUMMARY_TABLE_NAME, SUMMARIZED_TABLE_NAME) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_METADATA_SUMMARY_ENTRY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_STORE_ENTRY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index IX_JOB_STORE_ENTRY_WEU created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WN created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WS created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index IX_WORKFLOW_STORE_ENTRY_WS created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_again.xml::standardize_column_names_again::kshakir ran successfully in 127ms Running Changeset: changesets/rename_workflow_options_in_metadata.xml::rename_workflow_options_in_metadata::tjeandet [2024-01-10 13:58:49,92] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000 [2024-01-10 13:58:49,94] [info] [RenameWorkflowOptionsInMetadata] 100% Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: RenameWorkflowOptionsInMetadata complete. Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/rename_workflow_options_in_metadata.xml::rename_workflow_options_in_metadata::tjeandet ran successfully in 23ms Running Changeset: changesets/encrypt_and_clear_workflow_options.xml::encrypt_workflow_store_entry_workflow_options::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: EncryptWorkflowStoreEntryWorkflowOptions complete. Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/encrypt_and_clear_workflow_options.xml::encrypt_workflow_store_entry_workflow_options::kshakir ran successfully in 2ms Running Changeset: changesets/encrypt_and_clear_workflow_options.xml::clear_metadata_entry_workflow_options::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ClearMetadataEntryWorkflowOptions complete. Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/encrypt_and_clear_workflow_options.xml::clear_metadata_entry_workflow_options::kshakir ran successfully in 1ms Running Changeset: changesets/sub_workflow_store.xml::SUB_WORKFLOW_STORE_ENTRY::tjeandet Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Table SUB_WORKFLOW_STORE_ENTRY created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::SUB_WORKFLOW_STORE_ENTRY::tjeandet ran successfully in 2ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_uuid_index::tjeandet Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Index IX_SUB_WORKFLOW_STORE_ENTRY_PWEU created Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_uuid_index::tjeandet ran successfully in 2ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_jobkey_index::tjeandet Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to SUB_WORKFLOW_STORE_ENTRY(PARENT_WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, CALL_INDEX, CALL_ATTEMPT) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_jobkey_index::tjeandet ran successfully in 2ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_root_workflow_fk::tjeandet Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to SUB_WORKFLOW_STORE_ENTRY (ROOT_WORKFLOW_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_root_workflow_fk::tjeandet ran successfully in 2ms Running Changeset: changesets/workflow_store_imports_file.xml::workflow-store-imports-file::cjllanwarne Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Columns IMPORTS_ZIP(BLOB) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_imports_file.xml::workflow-store-imports-file::cjllanwarne ran successfully in 2ms Running Changeset: changesets/workflow_store_labels_file.xml::workflow-store-labels-file::cjllanwarne Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Columns CUSTOM_LABELS(LONGVARCHAR) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_labels_file.xml::workflow-store-labels-file::cjllanwarne ran successfully in 2ms Running Changeset: changesets/embiggen_detritus_value.xml::embiggen_call_caching_detritus_entry::rmunshi Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE datatype was changed to LONGVARCHAR Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/embiggen_detritus_value.xml::embiggen_call_caching_detritus_entry::rmunshi ran successfully in 2ms Running Changeset: changesets/standardize_column_names_patches.xml::standardize_column_names_patches::kshakir Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_HASH_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key FK_JOB_STORE_RESULT_SIMPLETON_JOB_STORE_ENTRY_ID dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key FK_SUB_WORKFLOW_STORE_ROOT_WORKFLOW_ID_WORKFLOW_STORE_ENTRY_ID dropped Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint UC_CALL_CACHING_ENTRY_WEU_CQFN_JI dropped from CALL_CACHING_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint UC_CALL_CACHING_HASH_ENTRY_CCEI dropped from CALL_CACHING_HASH_ENTRY Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_HASH_ENTRY(CALL_CACHING_ENTRY_ID, HASH_KEY) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: Foreign key constraint added to SUB_WORKFLOW_STORE_ENTRY (ROOT_WORKFLOW_ID) Jan 10, 2024 1:58:49 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_patches.xml::standardize_column_names_patches::kshakir ran successfully in 13ms Running Changeset: changesets/standardize_column_names_patches.xml::hsqldb_longvarchar_to_longtext::kshakir Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE datatype was changed to LONGTEXT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.SIMPLETON_VALUE datatype was changed to LONGTEXT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: JOB_STORE_ENTRY.EXCEPTION_MESSAGE datatype was changed to LONGTEXT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: JOB_STORE_SIMPLETON_ENTRY.SIMPLETON_VALUE datatype was changed to LONGTEXT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.CUSTOM_LABELS datatype was changed to LONGTEXT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_DEFINITION datatype was changed to LONGTEXT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_INPUTS datatype was changed to LONGTEXT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_OPTIONS datatype was changed to LONGTEXT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_patches.xml::hsqldb_longvarchar_to_longtext::kshakir ran successfully in 12ms Running Changeset: changesets/nullable_lobs.xml::nullable_lobs::kshakir Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Null constraint dropped from CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Null constraint dropped from CALL_CACHING_SIMPLETON_ENTRY.SIMPLETON_VALUE Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Null constraint dropped from JOB_STORE_SIMPLETON_ENTRY.SIMPLETON_VALUE Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_DEFINITION Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_INPUTS Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_OPTIONS Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.CUSTOM_LABELS Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/nullable_lobs.xml::nullable_lobs::kshakir ran successfully in 10ms Running Changeset: changesets/add_attempt_in_call_caching_entry.xml::add_attempt_in_call_caching_entry::tjeandet Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Columns JOB_ATTEMPT(INT) added to CALL_CACHING_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/add_attempt_in_call_caching_entry.xml::add_attempt_in_call_caching_entry::tjeandet ran successfully in 2ms Running Changeset: changesets/replace_empty_custom_labels.xml::custom_labels_not_null::rmunshi Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_STORE_ENTRY.CUSTOM_LABELS Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/replace_empty_custom_labels.xml::custom_labels_not_null::rmunshi ran successfully in 4ms Running Changeset: changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry::tjeandet Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table CALL_CACHING_AGGREGATION_ENTRY created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry::tjeandet ran successfully in 4ms Running Changeset: changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry_keys_and_indexes::tjeandet Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_AGGREGATION_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index IX_CALL_CACHING_AGGREGATION_ENTRY_BA_IFA created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry_keys_and_indexes::tjeandet ran successfully in 3ms Running Changeset: changesets/custom_label_entry.xml::custom_LABEL_entry::rmunshi Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::custom_LABEL_entry::rmunshi ran successfully in 2ms Running Changeset: changesets/custom_label_entry.xml::WMSE_workflow_execution_uuid_foreign_key::rmunshi Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Foreign key constraint added to CUSTOM_LABEL_ENTRY (WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::WMSE_workflow_execution_uuid_foreign_key::rmunshi ran successfully in 2ms Running Changeset: changesets/custom_label_entry.xml::custom_label_entry_index::rmunshi Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, CUSTOM_LABEL_VALUE, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::custom_label_entry_index::rmunshi ran successfully in 2ms Running Changeset: changesets/custom_label_entry.xml::modify label key/value data types::rmunshi Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_KEY datatype was changed to VARCHAR(63) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_VALUE datatype was changed to VARCHAR(63) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::modify label key/value data types::rmunshi ran successfully in 3ms Running Changeset: changesets/docker_hash_store.xml::DOCKER_HASH_STORE_ENTRY::tjeandet Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table DOCKER_HASH_STORE_ENTRY created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store.xml::DOCKER_HASH_STORE_ENTRY::tjeandet ran successfully in 3ms Running Changeset: changesets/docker_hash_store.xml::docker_hash_store_uuid_unique_constraint::tjeandet Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Unique constraint added to DOCKER_HASH_STORE_ENTRY(WORKFLOW_EXECUTION_UUID, DOCKER_TAG) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store.xml::docker_hash_store_uuid_unique_constraint::tjeandet ran successfully in 2ms Running Changeset: changesets/workflow_store_type_and_version.xml::workflow-store-type-and-version::mcovarr Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Columns WORKFLOW_TYPE(VARCHAR(30)),WORKFLOW_TYPE_VERSION(VARCHAR(255)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_type_and_version.xml::workflow-store-type-and-version::mcovarr ran successfully in 3ms Running Changeset: changesets/remove_pre_pbe_tables.xml::drop-pre-pbe-tables::mcovarr Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table EXECUTION_INFO dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table EXECUTION_EVENT dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table FAILURE_EVENT dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table RUNTIME_ATTRIBUTES dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table SYMBOL dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table EXECUTION dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table WORKFLOW_EXECUTION_AUX dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table WORKFLOW_EXECUTION dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/remove_pre_pbe_tables.xml::drop-pre-pbe-tables::mcovarr ran successfully in 14ms Running Changeset: changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Foreign key FK_CUSTOM_LABEL_ENTRY_WORKFLOW_EXECUTION_UUID dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Unique constraint UC_CUSTOM_LABEL_ENTRY_CLK_CLV_WEU dropped from CUSTOM_LABEL_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Unique constraint UC_SUMMARY_STATUS_ENTRY_STN_STN dropped from SUMMARY_STATUS_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Unique constraint UC_WORKFLOW_METADATA_SUMMARY_ENTRY_WEU dropped from WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table METADATA_ENTRY dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table SUMMARY_STATUS_ENTRY dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY_ENTRY dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir ran successfully in 12ms Running Changeset: changesets/workflow_store_state_widening.xml::workflow-store-state-widening::tjeandet Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_STATE datatype was changed to varchar(20) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_state_widening.xml::workflow-store-state-widening::tjeandet ran successfully in 2ms Running Changeset: changesets/workflow_store_restarted_column.xml::workflow-store-restarted-column::tjeandet Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Columns RESTARTED(BOOLEAN) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_restarted_column.xml::workflow-store-restarted-column::tjeandet ran successfully in 5ms Running Changeset: changesets/workflow_store_restarted_column.xml::update-restartable::tjeandet Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Data updated in WORKFLOW_STORE_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Data updated in WORKFLOW_STORE_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_restarted_column.xml::update-restartable::tjeandet ran successfully in 3ms Running Changeset: changesets/workflow_store_workflow_root_column.xml::workflow-store-workflow-root-column::tjeandet Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Columns WORKFLOW_ROOT(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_workflow_root_column.xml::workflow-store-workflow-root-column::tjeandet ran successfully in 2ms Running Changeset: changesets/workflow_store_horizontal_db.xml::workflow-store-horizontal-db::mcovarr Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Column WORKFLOW_STORE_ENTRY.RESTARTED dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Columns CROMWELL_ID(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Columns HEARTBEAT_TIMESTAMP(TIMESTAMP) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_horizontal_db.xml::workflow-store-horizontal-db::mcovarr ran successfully in 5ms Running Changeset: changesets/add_workflow_url_in_workflow_store_entry.xml::add_workflow_url_in_workflow_store_entry::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Columns WORKFLOW_URL(VARCHAR(255)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/add_workflow_url_in_workflow_store_entry.xml::add_workflow_url_in_workflow_store_entry::sshah ran successfully in 2ms Running Changeset: changesets/change_max_size_workflow_url.xml::change_max_size_for_workflow_url::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_URL datatype was changed to VARCHAR(2000) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/change_max_size_workflow_url.xml::change_max_size_for_workflow_url::sshah ran successfully in 1ms Running Changeset: changesets/docker_hash_store_add_size_column.xml::docker_hash_store_entry_add_size_column::tjeandet Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Columns DOCKER_SIZE(BIGINT) added to DOCKER_HASH_STORE_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store_add_size_column.xml::docker_hash_store_entry_add_size_column::tjeandet ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_hash_entry_id.xml::restore_auto_increment_call_caching_hash_entry_id::mcovarr Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_hash_entry_id.xml::restore_auto_increment_call_caching_hash_entry_id::mcovarr ran successfully in 2ms Running Changeset: changesets/add_hog_group_in_workflow_store.xml::add_hog_group_in_workflow_store::cjllanwarne Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Columns HOG_GROUP(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/add_hog_group_in_workflow_store.xml::add_hog_group_in_workflow_store::cjllanwarne ran successfully in 2ms Running Changeset: changesets/resync_engine_schema.xml::resync-engine-schema::kshakir Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.HEARTBEAT_TIMESTAMP datatype was changed to DATETIME(6) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.SUBMISSION_TIME datatype was changed to DATETIME(6) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_STORE_ENTRY.SUBMISSION_TIME Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/resync_engine_schema.xml::resync-engine-schema::kshakir ran successfully in 5ms Running Changeset: changesets/enlarge_job_store_ids.xml::drop_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Foreign key FK_JOB_STORE_SIMPLETON_ENTRY_JOB_STORE_ENTRY_ID dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::drop_job_store_simpleton_entry_fk::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_entry_id::mcovarr Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_ENTRY.JOB_STORE_ENTRY_ID Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_entry_id::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_id::mcovarr Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_SIMPLETON_ENTRY_ID Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_id::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_fk::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::recreate_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::recreate_job_store_simpleton_entry_fk::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_AGGREGATION_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_detritus_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_DETRITUS_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_detritus_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_hash_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_HASH_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_hash_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_SIMPLETON_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_cache_entry_id::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: CALL_CACHING_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_cache_entry_id::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::reset_call_cache_entry_id_autoincrement::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_ENTRY.CALL_CACHING_ENTRY_ID Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::reset_call_cache_entry_id_autoincrement::sshah ran successfully in 3ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_aggregation_entry_fk::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_aggregation_entry_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_detritus_entry_fk::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_detritus_entry_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_hash_entry_fk::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: CALL_CACHING_HASH_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_hash_entry_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_simpleton_entry_fk::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_simpleton_entry_fk::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_AGGREGATION_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_detritus_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_DETRITUS_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_detritus_entry_call_caching_entry_id_fk::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_hash_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_hash_entry_call_caching_entry_id_fk::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_SIMPLETON_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_aggregation_entry_id.xml::enlarge_call_cache_aggregation_entry_id::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_AGGREGATION_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_aggregation_entry_id.xml::enlarge_call_cache_aggregation_entry_id::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_aggregation_entry_id.xml::reset_call_caching_aggregation_entry_id_autoincrement::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_AGGREGATION_ENTRY_ID Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_aggregation_entry_id.xml::reset_call_caching_aggregation_entry_id_autoincrement::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_caching_detritus_entry_id.xml::enlarge_call_cache_detritus_entry_id::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_DETRITUS_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_detritus_entry_id.xml::enlarge_call_cache_detritus_entry_id::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_caching_detritus_entry_id.xml::reset_call_caching_detritus_entry_id_autoincrement::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_DETRITUS_ENTRY_ID Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_detritus_entry_id.xml::reset_call_caching_detritus_entry_id_autoincrement::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_simpleton_entry_id.xml::enlarge_call_cache_simpleton_entry_id::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_SIMPLETON_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_simpleton_entry_id.xml::enlarge_call_cache_simpleton_entry_id::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_simpleton_entry_id.xml::reset_call_caching_simpleton_entry_id_autoincrement::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_SIMPLETON_ENTRY_ID Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_simpleton_entry_id.xml::reset_call_caching_simpleton_entry_id_autoincrement::sshah ran successfully in 2ms Running Changeset: changesets/reset_call_caching_hash_entry_id_autoincrement.xml::reset_call_caching_hash_entry_id_autoincrement::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/reset_call_caching_hash_entry_id_autoincrement.xml::reset_call_caching_hash_entry_id_autoincrement::sshah ran successfully in 2ms Jan 10, 2024 1:58:50 PM liquibase.lockservice INFO: Successfully released change log lock [2024-01-10 13:58:50,34] [info] Running with database db.url = jdbc:hsqldb:mem:7bca2bab-e910-4d7f-8a21-537fad317b99;shutdown=false;hsqldb.tx=mvcc Jan 10, 2024 1:58:50 PM liquibase.database INFO: Set default schema name to PUBLIC Jan 10, 2024 1:58:50 PM liquibase.lockservice INFO: Successfully acquired change log lock Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Creating database history table with name: PUBLIC.SQLMETADATADATABASECHANGELOG Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Reading from PUBLIC.SQLMETADATADATABASECHANGELOG Running Changeset: metadata_changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table METADATA_ENTRY created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table SUMMARY_STATUS_ENTRY created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY_ENTRY created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, CUSTOM_LABEL_VALUE, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS_ENTRY(SUMMARY_TABLE_NAME, SUMMARIZED_TABLE_NAME) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_METADATA_SUMMARY_ENTRY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Foreign key constraint added to CUSTOM_LABEL_ENTRY (WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WN created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WS created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index METADATA_JOB_AND_KEY_IDX created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index METADATA_JOB_IDX created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index METADATA_WORKFLOW_IDX created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index SYS_IDX_11226 created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir ran successfully in 27ms Running Changeset: changesets/change_max_size_label_entry.xml::change_max_size_for_key_and_value_in_label::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_KEY datatype was changed to VARCHAR(255) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_VALUE datatype was changed to VARCHAR(255) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet changesets/change_max_size_label_entry.xml::change_max_size_for_key_and_value_in_label::sshah ran successfully in 3ms Running Changeset: metadata_changesets/delete_duplicate_custom_labels.xml::delete_duplicate_custom_labels::kshakir Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Data deleted from CUSTOM_LABEL_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Unique constraint UC_CUSTOM_LABEL_ENTRY_CLK_CLV_WEU dropped from CUSTOM_LABEL_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/delete_duplicate_custom_labels.xml::delete_duplicate_custom_labels::kshakir ran successfully in 6ms Running Changeset: metadata_changesets/add_submission_timestamp_metadata_summary.xml::add_submission_timestamp_in_metadata_entry::sshah Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Columns SUBMISSION_TIMESTAMP(DATETIME(6)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_submission_timestamp_metadata_summary.xml::add_submission_timestamp_in_metadata_entry::sshah ran successfully in 2ms Running Changeset: metadata_changesets/custom_label_entry_key_value_index.xml::custom_label_entry_key_value_index::mcovarr Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index IX_CUSTOM_LABEL_ENTRY_CLK_CLV created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/custom_label_entry_key_value_index.xml::custom_label_entry_key_value_index::mcovarr ran successfully in 2ms Running Changeset: metadata_changesets/add_parent_and_root_workflow_execution_uuid.xml::add_parent_and_root_workflow_execution_uuid::kshakir Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Columns PARENT_WORKFLOW_EXECUTION_UUID(VARCHAR(100)),ROOT_WORKFLOW_EXECUTION_UUID(VARCHAR(100)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_PWEU created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_RWEU created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Unique constraint UC_SUMMARY_STATUS_ENTRY_STN_STN dropped from SUMMARY_STATUS_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.SUMMARY_TABLE_NAME renamed to SUMMARY_NAME Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.SUMMARIZED_TABLE_NAME dropped Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.MAXIMUM_ID renamed to SUMMARY_POSITION Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS_ENTRY(SUMMARY_NAME) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_parent_and_root_workflow_execution_uuid.xml::add_parent_and_root_workflow_execution_uuid::kshakir ran successfully in 13ms Running Changeset: metadata_changesets/add_parent_and_root_workflow_typo_fix.xml::add_parent_and_root_workflow_typo_fix::kshakir Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Data deleted from SUMMARY_STATUS_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_parent_and_root_workflow_typo_fix.xml::add_parent_and_root_workflow_typo_fix::kshakir ran successfully in 3ms Running Changeset: metadata_changesets/metadata_index_removals.xml::metadata_index_removals::mcovarr Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index METADATA_JOB_AND_KEY_IDX dropped from table METADATA_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index METADATA_JOB_IDX dropped from table METADATA_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/metadata_index_removals.xml::metadata_index_removals::mcovarr ran successfully in 3ms Running Changeset: metadata_changesets/add_metadata_archive_status.xml::add_metadata_archive_status::cjllanwarne Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Columns METADATA_ARCHIVE_STATUS(VARCHAR(30)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_metadata_archive_status.xml::add_metadata_archive_status::cjllanwarne ran successfully in 2ms Running Changeset: metadata_changesets/add_metadata_archive_status.xml::metadata_archive_status_index::cjllanwarne Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_metadata_archive_status.xml::metadata_archive_status_index::cjllanwarne ran successfully in 1ms Running Changeset: metadata_changesets/summarization_queue_table.xml::summary_queue_table::gsterin Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Table SUMMARY_QUEUE_ENTRY created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table.xml::summary_queue_table::gsterin ran successfully in 2ms Running Changeset: metadata_changesets/summarization_queue_table.xml::summary_queue_initialization::gsterin Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table.xml::summary_queue_initialization::gsterin ran successfully in 4ms Running Changeset: metadata_changesets/summarization_queue_table_add_primary_key.xml::summary_queue_add_primary_key::gsterin Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Primary key added to SUMMARY_QUEUE_ENTRY (METADATA_JOURNAL_ID) Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table_add_primary_key.xml::summary_queue_add_primary_key::gsterin ran successfully in 1ms Running Changeset: metadata_changesets/remove_non_summarizable_metadata_from_queue.xml::delete_non_summarizable_metadata_from_queue::mcovarr Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/remove_non_summarizable_metadata_from_queue.xml::delete_non_summarizable_metadata_from_queue::mcovarr ran successfully in 3ms Running Changeset: metadata_changesets/update_metadata_archive_index.xml::update_old_metadata_archive_status_index::cjllanwarne Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS dropped from table WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS_ET created Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/update_metadata_archive_index.xml::update_old_metadata_archive_status_index::cjllanwarne ran successfully in 3ms Running Changeset: metadata_changesets/reset_archive_statuses_to_null.xml::reset_archive_statuses_to_null::cjllanwarne Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: Data updated in WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 1:58:50 PM liquibase.changelog INFO: ChangeSet metadata_changesets/reset_archive_statuses_to_null.xml::reset_archive_statuses_to_null::cjllanwarne ran successfully in 2ms Jan 10, 2024 1:58:50 PM liquibase.lockservice INFO: Successfully released change log lock [2024-01-10 13:58:51,01] [info] Slf4jLogger started [2024-01-10 13:58:51,23] [info] Workflow heartbeat configuration: { "cromwellId" : "cromid-fa283e9", "heartbeatInterval" : "2 minutes", "ttl" : "10 minutes", "failureShutdownDuration" : "5 minutes", "writeBatchSize" : 10000, "writeThreshold" : 10000 } [2024-01-10 13:58:51,31] [info] Metadata summary refreshing every 1 second. [2024-01-10 13:58:51,31] [info] No metadata archiver defined in config [2024-01-10 13:58:51,32] [info] No metadata deleter defined in config [2024-01-10 13:58:51,32] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds. [2024-01-10 13:58:51,32] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds. [2024-01-10 13:58:51,38] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds. [2024-01-10 13:58:51,46] [info] JobRestartCheckTokenDispenser - Distribution rate: 50 per 1 seconds. [2024-01-10 13:58:51,51] [info] JobExecutionTokenDispenser - Distribution rate: 20 per 10 seconds. [2024-01-10 13:58:51,55] [info] SingleWorkflowRunnerActor: Version 83 [2024-01-10 13:58:51,57] [info] SingleWorkflowRunnerActor: Submitting workflow [2024-01-10 13:58:51,64] [info] Unspecified type (Unspecified version) workflow 7958a2eb-d020-46f1-83d0-6873d5a32972 submitted [2024-01-10 13:58:51,67] [info] SingleWorkflowRunnerActor: Workflow submitted 7958a2eb-d020-46f1-83d0-6873d5a32972 [2024-01-10 13:58:51,68] [info] 1 new workflows fetched by cromid-fa283e9: 7958a2eb-d020-46f1-83d0-6873d5a32972 [2024-01-10 13:58:51,69] [info] WorkflowManagerActor: Starting workflow 7958a2eb-d020-46f1-83d0-6873d5a32972 [2024-01-10 13:58:51,70] [info] WorkflowManagerActor: Successfully started WorkflowActor-7958a2eb-d020-46f1-83d0-6873d5a32972 [2024-01-10 13:58:51,70] [info] Retrieved 1 workflows from the WorkflowStoreActor [2024-01-10 13:58:51,74] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes. [2024-01-10 13:58:51,85] [info] MaterializeWorkflowDescriptorActor [7958a2eb]: Parsing workflow as WDL 1.0 [2024-01-10 13:58:53,48] [info] MaterializeWorkflowDescriptorActor [7958a2eb]: Call-to-Backend assignments: theiacov_clearlabs.pangolin4 -> Local, theiacov_clearlabs.qc_check_task -> Local, theiacov_clearlabs.sc2_gene_coverage -> Local, theiacov_clearlabs.nextclade -> Local, theiacov_clearlabs.kraken2_dehosted -> Local, theiacov_clearlabs.consensus -> Local, theiacov_clearlabs.consensus_qc -> Local, theiacov_clearlabs.version_capture -> Local, theiacov_clearlabs.kraken2_raw -> Local, theiacov_clearlabs.nextclade_output_parser -> Local, theiacov_clearlabs.fastq_scan_raw_reads -> Local, theiacov_clearlabs.vadr -> Local, theiacov_clearlabs.stats_n_coverage_primtrim -> Local, theiacov_clearlabs.stats_n_coverage -> Local, theiacov_clearlabs.fastq_scan_clean_reads -> Local, theiacov_clearlabs.ncbi_scrub_se -> Local [2024-01-10 13:58:53,72] [warn] Local [7958a2eb]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,72] [warn] Local [7958a2eb]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,72] [warn] Local [7958a2eb]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,72] [warn] Local [7958a2eb]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,72] [warn] Local [7958a2eb]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,72] [warn] Local [7958a2eb]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,72] [warn] Local [7958a2eb]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,73] [warn] Local [7958a2eb]: Key/s [dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,73] [warn] Local [7958a2eb]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,73] [warn] Local [7958a2eb]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,73] [warn] Local [7958a2eb]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,73] [warn] Local [7958a2eb]: Key/s [dx_instance_type, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,73] [warn] Local [7958a2eb]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,73] [warn] Local [7958a2eb]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,73] [warn] Local [7958a2eb]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:53,73] [warn] Local [7958a2eb]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 13:58:54,83] [info] WorkflowExecutionActor-7958a2eb-d020-46f1-83d0-6873d5a32972 [7958a2eb]: Starting theiacov_clearlabs.version_capture [2024-01-10 13:58:55,91] [info] WorkflowExecutionActor-7958a2eb-d020-46f1-83d0-6873d5a32972 [7958a2eb]: Starting theiacov_clearlabs.fastq_scan_raw_reads, theiacov_clearlabs.ncbi_scrub_se, theiacov_clearlabs.kraken2_raw [2024-01-10 13:58:56,48] [info] Not triggering log of restart checking token queue status. Effective log interval = None [2024-01-10 13:58:56,52] [info] Not triggering log of execution token queue status. Effective log interval = None [2024-01-10 13:59:01,54] [info] Assigned new job execution tokens to the following groups: 7958a2eb: 4 [2024-01-10 13:59:02,54] [warn] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.ncbi_scrub_se:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 13:59:02,54] [warn] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.fastq_scan_raw_reads:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 13:59:02,54] [warn] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.version_capture:NA:1]: Unrecognized runtime attribute keys: dx_instance_type, disks, cpu, memory [2024-01-10 13:59:02,61] [warn] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.kraken2_raw:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 13:59:02,64] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.version_capture:NA:1]: PHB_Version="PHB v1.2.1" date +"%Y-%m-%d" > TODAY echo "$PHB_Version" > PHB_VERSION [2024-01-10 13:59:02,64] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.ncbi_scrub_se:NA:1]: # date and version control date | tee DATE # unzip fwd file as scrub tool does not take in .gz fastq files if [[ "/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-ncbi_scrub_se/inputs/-1726380058/clearlabs.fastq.gz" == *.gz ]] then gunzip -c /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-ncbi_scrub_se/inputs/-1726380058/clearlabs.fastq.gz > r1.fastq read1_unzip=r1.fastq else read1_unzip=/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-ncbi_scrub_se/inputs/-1726380058/clearlabs.fastq.gz fi # dehost reads /opt/scrubber/scripts/scrub.sh -n ${read1_unzip} |& tail -n1 | awk -F" " '{print $1}' > FWD_SPOTS_REMOVED # gzip dehosted reads gzip ${read1_unzip}.clean -c > clearlabs_R1_dehosted.fastq.gz [2024-01-10 13:59:02,64] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.fastq_scan_raw_reads:NA:1]: # capture date and version date | tee DATE fastq-scan -v | tee VERSION # set cat command based on compression if [[ "/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_raw_reads/inputs/-1726380058/clearlabs.fastq.gz" == *".gz" ]] ; then cat_reads="zcat" else cat_reads="cat" fi # capture forward read stats eval "${cat_reads} /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_raw_reads/inputs/-1726380058/clearlabs.fastq.gz" | fastq-scan | tee clearlabs_fastq-scan.json cat clearlabs_fastq-scan.json | jq .qc_stats.read_total | tee READ1_SEQS [2024-01-10 13:59:02,64] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.kraken2_raw:NA:1]: # date and version control date | tee DATE kraken2 --version | head -n1 | tee VERSION num_reads=$(ls *fastq.gz 2> /dev/nul | wc -l) if ! [ -z ]; then mode="--paired" fi echo $mode kraken2 $mode \ --threads 4 \ --db /kraken2-db \ /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_raw/inputs/-1726380058/clearlabs.fastq.gz \ --report clearlabs_kraken2_report.txt >/dev/null percentage_human=$(grep "Homo sapiens" clearlabs_kraken2_report.txt | cut -f 1) # | tee PERCENT_HUMAN percentage_sc2=$(grep "Severe acute respiratory syndrome coronavirus 2" clearlabs_kraken2_report.txt | cut -f1 ) # | tee PERCENT_COV if [ -z "$percentage_human" ] ; then percentage_human="0" ; fi if [ -z "$percentage_sc2" ] ; then percentage_sc2="0" ; fi echo $percentage_human | tee PERCENT_HUMAN echo $percentage_sc2 | tee PERCENT_SC2 # capture target org percentage if [ ! -z "" ]; then echo "Target org designated: " percent_target_org=$(grep "" clearlabs_kraken2_report.txt | cut -f1 | head -n1 ) if [-z "$percent_target_org" ] ; then percent_target_org="0" ; fi else percent_target_org="" fi echo $percent_target_org | tee PERCENT_TARGET_ORG [2024-01-10 13:59:02,77] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.version_capture:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-version_capture/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-version_capture/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-version_capture:/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-version_capture:delegated \ us-docker.pkg.dev/general-theiagen/ubuntu/ubuntu@sha256:9a35cef02b57290a7c919824b632897fbe6db3aa18c930110b04d81fcad458ff /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-version_capture/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-version_capture/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-version_capture/execution/docker_cid # return exit code exit $rc [2024-01-10 13:59:02,77] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.ncbi_scrub_se:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-ncbi_scrub_se/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-ncbi_scrub_se/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-ncbi_scrub_se:/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-ncbi_scrub_se:delegated \ us-docker.pkg.dev/general-theiagen/ncbi/sra-human-scrubber@sha256:b7dba71079344daea4ea3363e1a67fa54edb7ec65459d039669c68a66d38b140 /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-ncbi_scrub_se/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-ncbi_scrub_se/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-ncbi_scrub_se/execution/docker_cid # return exit code exit $rc [2024-01-10 13:59:02,77] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.kraken2_raw:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_raw/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_raw/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_raw:/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_raw:delegated \ us-docker.pkg.dev/general-theiagen/staphb/kraken2@sha256:7884b5a6ec3c20cd7edbd42b92ae7e9806b4581997acb270b1b9ecb627474e2a /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_raw/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_raw/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_raw/execution/docker_cid # return exit code exit $rc [2024-01-10 13:59:02,77] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.fastq_scan_raw_reads:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_raw_reads/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_raw_reads/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_raw_reads:/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_raw_reads:delegated \ us-docker.pkg.dev/general-theiagen/biocontainers/fastq-scan@sha256:dd4880ca592ef724a15cd63dfcb4e3e04f5d4b269976e563ab7413d8ca5cfcfa /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_raw_reads/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_raw_reads/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_raw_reads/execution/docker_cid # return exit code exit $rc [2024-01-10 13:59:06,36] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.kraken2_raw:NA:1]: job id: 44196 [2024-01-10 13:59:06,36] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.fastq_scan_raw_reads:NA:1]: job id: 44197 [2024-01-10 13:59:06,36] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.ncbi_scrub_se:NA:1]: job id: 44198 [2024-01-10 13:59:06,36] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.version_capture:NA:1]: job id: 44202 [2024-01-10 13:59:06,37] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.ncbi_scrub_se:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 13:59:06,37] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.version_capture:NA:1]: Status change from - to Done [2024-01-10 13:59:06,37] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.fastq_scan_raw_reads:NA:1]: Status change from - to Done [2024-01-10 13:59:06,37] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.kraken2_raw:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 13:59:13,60] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.ncbi_scrub_se:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 13:59:15,32] [info] WorkflowExecutionActor-7958a2eb-d020-46f1-83d0-6873d5a32972 [7958a2eb]: Starting theiacov_clearlabs.kraken2_dehosted, theiacov_clearlabs.consensus, theiacov_clearlabs.fastq_scan_clean_reads [2024-01-10 13:59:21,53] [info] Assigned new job execution tokens to the following groups: 7958a2eb: 3 [2024-01-10 13:59:21,54] [warn] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.kraken2_dehosted:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 13:59:21,54] [warn] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.fastq_scan_clean_reads:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 13:59:21,54] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_dehosted/inputs/297104305/clearlabs_R1_dehosted.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-ncbi_scrub_se/execution/clearlabs_R1_dehosted.fastq.gz: Operation not permitted [2024-01-10 13:59:21,55] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_clean_reads/inputs/297104305/clearlabs_R1_dehosted.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-ncbi_scrub_se/execution/clearlabs_R1_dehosted.fastq.gz: Operation not permitted [2024-01-10 13:59:21,56] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.kraken2_dehosted:NA:1]: # date and version control date | tee DATE kraken2 --version | head -n1 | tee VERSION num_reads=$(ls *fastq.gz 2> /dev/nul | wc -l) if ! [ -z ]; then mode="--paired" fi echo $mode kraken2 $mode \ --threads 4 \ --db /kraken2-db \ /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_dehosted/inputs/297104305/clearlabs_R1_dehosted.fastq.gz \ --report clearlabs_kraken2_report.txt >/dev/null percentage_human=$(grep "Homo sapiens" clearlabs_kraken2_report.txt | cut -f 1) # | tee PERCENT_HUMAN percentage_sc2=$(grep "Severe acute respiratory syndrome coronavirus 2" clearlabs_kraken2_report.txt | cut -f1 ) # | tee PERCENT_COV if [ -z "$percentage_human" ] ; then percentage_human="0" ; fi if [ -z "$percentage_sc2" ] ; then percentage_sc2="0" ; fi echo $percentage_human | tee PERCENT_HUMAN echo $percentage_sc2 | tee PERCENT_SC2 # capture target org percentage if [ ! -z "" ]; then echo "Target org designated: " percent_target_org=$(grep "" clearlabs_kraken2_report.txt | cut -f1 | head -n1 ) if [-z "$percent_target_org" ] ; then percent_target_org="0" ; fi else percent_target_org="" fi echo $percent_target_org | tee PERCENT_TARGET_ORG [2024-01-10 13:59:21,57] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.fastq_scan_clean_reads:NA:1]: # capture date and version date | tee DATE fastq-scan -v | tee VERSION # set cat command based on compression if [[ "/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_clean_reads/inputs/297104305/clearlabs_R1_dehosted.fastq.gz" == *".gz" ]] ; then cat_reads="zcat" else cat_reads="cat" fi # capture forward read stats eval "${cat_reads} /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_clean_reads/inputs/297104305/clearlabs_R1_dehosted.fastq.gz" | fastq-scan | tee clearlabs_R1_dehosted_fastq-scan.json cat clearlabs_R1_dehosted_fastq-scan.json | jq .qc_stats.read_total | tee READ1_SEQS [2024-01-10 13:59:21,58] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.fastq_scan_clean_reads:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_clean_reads/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_clean_reads/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_clean_reads:/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_clean_reads:delegated \ us-docker.pkg.dev/general-theiagen/biocontainers/fastq-scan@sha256:dd4880ca592ef724a15cd63dfcb4e3e04f5d4b269976e563ab7413d8ca5cfcfa /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_clean_reads/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_clean_reads/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-fastq_scan_clean_reads/execution/docker_cid # return exit code exit $rc [2024-01-10 13:59:21,59] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.kraken2_dehosted:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_dehosted/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_dehosted/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_dehosted:/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_dehosted:delegated \ us-docker.pkg.dev/general-theiagen/staphb/kraken2@sha256:7884b5a6ec3c20cd7edbd42b92ae7e9806b4581997acb270b1b9ecb627474e2a /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_dehosted/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_dehosted/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-kraken2_dehosted/execution/docker_cid # return exit code exit $rc [2024-01-10 13:59:21,87] [warn] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.consensus:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 13:59:21,88] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/inputs/297104305/clearlabs_R1_dehosted.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-ncbi_scrub_se/execution/clearlabs_R1_dehosted.fastq.gz: Operation not permitted [2024-01-10 13:59:21,90] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.consensus:NA:1]: # HIV if [[ sars-cov-2 == "HIV" ]]; then # setup custom primer scheme (/V is required by Artic) mkdir -p ./primer-schemes/HIV/Vuser ## set reference genome ref_genome="" head -n1 "${ref_genome}" | sed 's/>//' | tee REFERENCE_GENOME cp "${ref_genome}" ./primer-schemes/HIV/Vuser/HIV.reference.fasta ## set primers #cp /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/inputs/1063836572/artic-v3.primers.bed ./primer-schemes/SARS-CoV-2/Vuser/SARS-CoV-2.scheme.bed #p_bed="/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/inputs/1063836572/artic-v3.primers.bed" cp "/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/inputs/1063836572/artic-v3.primers.bed" ./primer-schemes/HIV/Vuser/HIV.scheme.bed scheme_name="HIV/Vuser" # Add other viruses here # Default is SARS-CoV-2 else # setup custom primer scheme (/V is required by Artic) mkdir -p ./primer-schemes/SARS-CoV-2/Vuser ## set reference genome if [[ ! -z "" ]]; then ref_genome="" else # use reference file in docker--different paths depending on image specified if [[ -d "/fieldbioinformatics" ]]; then ref_genome=$(find /fieldbioinformatics/*/primer*schemes/nCoV-2019/V3/ -name "nCoV-2019.reference.fasta") else ref_genome=$(find /wf-artic*/data/primer_schemes/SARS-CoV-2/V4/ -name "SARS-CoV-2.reference.fasta") fi echo "No user-defined reference genome; setting reference to ${ref_genome}" fi head -n1 "${ref_genome}" | sed 's/>//' | tee REFERENCE_GENOME cp "${ref_genome}" ./primer-schemes/SARS-CoV-2/Vuser/SARS-CoV-2.reference.fasta ## set primers cp /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/inputs/1063836572/artic-v3.primers.bed ./primer-schemes/SARS-CoV-2/Vuser/SARS-CoV-2.scheme.bed scheme_name="SARS-CoV-2/Vuser" fi # version control echo "Medaka via $(artic -v)" | tee VERSION echo "artic-v3.primers.bed" | tee PRIMER_NAME artic minion --medaka --medaka-model r941_min_high_g360 --normalise 20000 --threads 8 --scheme-directory ./primer-schemes --read-file /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/inputs/297104305/clearlabs_R1_dehosted.fastq.gz ${scheme_name} clearlabs gunzip -f clearlabs.pass.vcf.gz # clean up fasta header echo ">clearlabs" > clearlabs.medaka.consensus.fasta grep -v ">" clearlabs.consensus.fasta >> clearlabs.medaka.consensus.fasta # grab reads from alignment samtools fastq -F4 clearlabs.primertrimmed.rg.sorted.bam | gzip > clearlabs.fastq.gz [2024-01-10 13:59:21,93] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.consensus:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus:/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus:delegated \ us-docker.pkg.dev/general-theiagen/staphb/artic-ncov2019@sha256:ee7b325330acc5cb738efcdb267d558ac0ca6a03ba22dc46393c3168a5469b56 /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/execution/docker_cid # return exit code exit $rc [2024-01-10 13:59:26,34] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.kraken2_dehosted:NA:1]: job id: 44857 [2024-01-10 13:59:26,34] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.fastq_scan_clean_reads:NA:1]: job id: 44852 [2024-01-10 13:59:26,34] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.consensus:NA:1]: job id: 44933 [2024-01-10 13:59:26,34] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.consensus:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 13:59:26,34] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.fastq_scan_clean_reads:NA:1]: Status change from - to Done [2024-01-10 13:59:26,35] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.kraken2_dehosted:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 13:59:28,01] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.kraken2_dehosted:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 13:59:30,78] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.kraken2_raw:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:02:57,08] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.consensus:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:02:59,11] [info] WorkflowExecutionActor-7958a2eb-d020-46f1-83d0-6873d5a32972 [7958a2eb]: Starting theiacov_clearlabs.pangolin4, theiacov_clearlabs.stats_n_coverage_primtrim, theiacov_clearlabs.sc2_gene_coverage, theiacov_clearlabs.nextclade, theiacov_clearlabs.stats_n_coverage, theiacov_clearlabs.consensus_qc [2024-01-10 14:03:01,53] [info] Assigned new job execution tokens to the following groups: 7958a2eb: 6 [2024-01-10 14:03:01,73] [warn] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.pangolin4:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:03:01,74] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-pangolin4/inputs/-1423075165/clearlabs.medaka.consensus.fasta -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/execution/clearlabs.medaka.consensus.fasta: Operation not permitted [2024-01-10 14:03:01,74] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.pangolin4:NA:1]: set -e # date and version capture date | tee DATE { pangolin --all-versions && usher --version; } | tr '\n' ';' | cut -f -6 -d ';' | tee VERSION_PANGOLIN_ALL pangolin "/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-pangolin4/inputs/-1423075165/clearlabs.medaka.consensus.fasta" \ \ --min-length 10000 \ --max-ambig 0.5 \ --expanded-lineage \ \ \ --outfile "clearlabs.pangolin_report.csv" \ --verbose \ python3 < /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/execution/clearlabs.primertrimmed.rg.sorted.bam: Operation not permitted [2024-01-10 14:03:01,85] [warn] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.stats_n_coverage:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:03:01,86] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage_primtrim/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/execution/clearlabs.primertrimmed.rg.sorted.bam: Operation not permitted [2024-01-10 14:03:01,86] [warn] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.consensus_qc:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:03:01,86] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage/inputs/-1423075165/clearlabs.trimmed.rg.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/execution/clearlabs.trimmed.rg.sorted.bam: Operation not permitted [2024-01-10 14:03:01,86] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus_qc/inputs/-1423075165/clearlabs.medaka.consensus.fasta -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/execution/clearlabs.medaka.consensus.fasta: Operation not permitted [2024-01-10 14:03:01,87] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.consensus_qc:NA:1]: if [ -s "" ] ; then GENOME_LEN=$(grep -v ">" | tr --delete '\n' | wc -c) elif [ ] ; then GENOME_LEN= else # set SC2 default GENOME_LEN=29903 fi # capture date and version date | tee DATE num_N=$( grep -v ">" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus_qc/inputs/-1423075165/clearlabs.medaka.consensus.fasta | grep -o 'N' | wc -l ) if [ -z "$num_N" ] ; then num_N="0" ; fi echo $num_N | tee NUM_N num_ACTG=$( grep -v ">" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus_qc/inputs/-1423075165/clearlabs.medaka.consensus.fasta | grep -o -E "C|A|T|G" | wc -l ) if [ -z "$num_ACTG" ] ; then num_ACTG="0" ; fi echo $num_ACTG | tee NUM_ACTG # calculate percent coverage (Wu Han-1 genome length: 29903bp) python3 -c "print ( round( ($num_ACTG / $GENOME_LEN ) * 100, 2 ) )" | tee PERCENT_REF_COVERAGE num_degenerate=$( grep -v ">" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus_qc/inputs/-1423075165/clearlabs.medaka.consensus.fasta | grep -o -E "B|D|E|F|H|I|J|K|L|M|O|P|Q|R|S|U|V|W|X|Y|Z" | wc -l ) if [ -z "$num_degenerate" ] ; then num_degenerate="0" ; fi echo $num_degenerate | tee NUM_DEGENERATE num_total=$( grep -v ">" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus_qc/inputs/-1423075165/clearlabs.medaka.consensus.fasta | grep -o -E '[A-Z]' | wc -l ) if [ -z "$num_total" ] ; then num_total="0" ; fi echo $num_total | tee NUM_TOTAL [2024-01-10 14:03:01,90] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.stats_n_coverage_primtrim:NA:1]: date | tee DATE samtools --version | head -n1 | tee VERSION samtools stats /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage_primtrim/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam > clearlabs.stats.txt samtools coverage /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage_primtrim/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam -m -o clearlabs.cov.hist samtools coverage /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage_primtrim/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam -o clearlabs.cov.txt samtools flagstat /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage_primtrim/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam > clearlabs.flagstat.txt coverage=$(cut -f 6 clearlabs.cov.txt | tail -n 1) depth=$(cut -f 7 clearlabs.cov.txt | tail -n 1) meanbaseq=$(cut -f 8 clearlabs.cov.txt | tail -n 1) meanmapq=$(cut -f 9 clearlabs.cov.txt | tail -n 1) if [ -z "$coverage" ] ; then coverage="0" ; fi if [ -z "$depth" ] ; then depth="0" ; fi if [ -z "$meanbaseq" ] ; then meanbaseq="0" ; fi if [ -z "$meanmapq" ] ; then meanmapq="0" ; fi echo $coverage | tee COVERAGE echo $depth | tee DEPTH echo $meanbaseq | tee MEANBASEQ echo $meanmapq | tee MEANMAPQ [2024-01-10 14:03:01,90] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.stats_n_coverage:NA:1]: date | tee DATE samtools --version | head -n1 | tee VERSION samtools stats /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage/inputs/-1423075165/clearlabs.trimmed.rg.sorted.bam > clearlabs.stats.txt samtools coverage /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage/inputs/-1423075165/clearlabs.trimmed.rg.sorted.bam -m -o clearlabs.cov.hist samtools coverage /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage/inputs/-1423075165/clearlabs.trimmed.rg.sorted.bam -o clearlabs.cov.txt samtools flagstat /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage/inputs/-1423075165/clearlabs.trimmed.rg.sorted.bam > clearlabs.flagstat.txt coverage=$(cut -f 6 clearlabs.cov.txt | tail -n 1) depth=$(cut -f 7 clearlabs.cov.txt | tail -n 1) meanbaseq=$(cut -f 8 clearlabs.cov.txt | tail -n 1) meanmapq=$(cut -f 9 clearlabs.cov.txt | tail -n 1) if [ -z "$coverage" ] ; then coverage="0" ; fi if [ -z "$depth" ] ; then depth="0" ; fi if [ -z "$meanbaseq" ] ; then meanbaseq="0" ; fi if [ -z "$meanmapq" ] ; then meanmapq="0" ; fi echo $coverage | tee COVERAGE echo $depth | tee DEPTH echo $meanbaseq | tee MEANBASEQ echo $meanmapq | tee MEANMAPQ [2024-01-10 14:03:01,90] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.sc2_gene_coverage:NA:1]: samtools index /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam chr=$(samtools idxstats /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam | cut -f 1 | head -1) samtools coverage -r "${chr}:21563-25384" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam >> clearlabs.cov.txt s_gene_depth=$(cut -f 7 clearlabs.cov.txt | tail -n 1) # samtools outputs 3 columns; column 3 is the depth of coverage per nucleotide position, piped to awk to count the positions # above min_depth, then wc -l counts them all orf1ab=$(samtools depth -J -r "${chr}:266-21555" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) sgene=$(samtools depth -J -r "${chr}:21563-25384" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) orf3a=$(samtools depth -J -r "${chr}:25393-26220" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) egene=$(samtools depth -J -r "${chr}:26245-26472" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) mgene=$(samtools depth -J -r "${chr}:26523-27191" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) orf6=$(samtools depth -J -r "${chr}:27202-27387" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) orf7a=$(samtools depth -J -r "${chr}:27394-27759" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) orf7b=$(samtools depth -J -r "${chr}:27756-27887" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) orf8=$(samtools depth -J -r "${chr}:27894-28259" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) ngene=$(samtools depth -J -r "${chr}:28274-29533" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) orf10=$(samtools depth -J -r "${chr}:29558-29674" /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/inputs/-1423075165/clearlabs.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) orf1ab_pc=$(python3 -c "print ( round( ($orf1ab / 21290 ) * 100, 2 ) )") sgene_pc=$(python3 -c "print ( round( ($sgene / 3822 ) * 100, 2 ) )") orf3a_pc=$(python3 -c "print ( round( ($orf3a / 828 ) * 100, 2 ) )") egene_pc=$(python3 -c "print ( round( ($egene / 228 ) * 100, 2 ) )") mgene_pc=$(python3 -c "print ( round( ($mgene / 669 ) * 100, 2 ) )") orf6_pc=$(python3 -c "print ( round( ($orf6 / 186 ) * 100, 2 ) )") orf7a_pc=$(python3 -c "print ( round( ($orf7a / 366 ) * 100, 2 ) )") orf7b_pc=$(python3 -c "print ( round( ($orf7b / 132 ) * 100, 2 ) )") orf8_pc=$(python3 -c "print ( round( ($orf8 / 366 ) * 100, 2 ) )") ngene_pc=$(python3 -c "print ( round( ($ngene / 1260 ) * 100, 2 ) )") orf10_pc=$(python3 -c "print ( round( ($orf10 / 117 ) * 100, 2 ) )") echo -e "#NOTE: THE VALUES BELOW ASSUME WUHAN-1 REFERENCE GENOME" > clearlabs.percent_gene_coverage.tsv echo -e "Gene\tPercent_Coverage" >> clearlabs.percent_gene_coverage.tsv echo -e "ORF1ab\t" $orf1ab_pc >> clearlabs.percent_gene_coverage.tsv echo -e "S_gene\t" $sgene_pc >> clearlabs.percent_gene_coverage.tsv echo -e "ORF3a\t" $orf3a_pc >> clearlabs.percent_gene_coverage.tsv echo -e "E_gene\t" $egene_pc >> clearlabs.percent_gene_coverage.tsv echo -e "M_gene\t" $mgene_pc >> clearlabs.percent_gene_coverage.tsv echo -e "ORF6\t" $orf6_pc >> clearlabs.percent_gene_coverage.tsv echo -e "ORF7a\t" $orf7a_pc >> clearlabs.percent_gene_coverage.tsv echo -e "ORF7b\t" $orf7b_pc >> clearlabs.percent_gene_coverage.tsv echo -e "ORF8\t" $orf8_pc >> clearlabs.percent_gene_coverage.tsv echo -e "N_gene\t" $ngene_pc >> clearlabs.percent_gene_coverage.tsv echo -e "ORF10\t" $orf10_pc >> clearlabs.percent_gene_coverage.tsv if [ -z "s_gene_depth" ] ; then s_gene_depth="0"; fi echo $s_gene_depth | tee S_GENE_DEPTH echo $sgene_pc | tee S_GENE_PC [2024-01-10 14:03:01,90] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.pangolin4:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-pangolin4/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-pangolin4/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-pangolin4:/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-pangolin4:delegated \ us-docker.pkg.dev/general-theiagen/staphb/pangolin@sha256:54b03e235ee04f007417cb934d1a263ce73bbd16aa82134efa680bd5d856c38d /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-pangolin4/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-pangolin4/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-pangolin4/execution/docker_cid # return exit code exit $rc [2024-01-10 14:03:01,93] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.sc2_gene_coverage:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage:/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage:delegated \ us-docker.pkg.dev/general-theiagen/staphb/samtools@sha256:ef92cb5e490c75e66e89ab0a113865839e6954435d363af400eaa27177ff7c91 /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-sc2_gene_coverage/execution/docker_cid # return exit code exit $rc [2024-01-10 14:03:01,93] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.consensus_qc:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus_qc/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus_qc/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus_qc:/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus_qc:delegated \ us-docker.pkg.dev/general-theiagen/theiagen/utility@sha256:b40130a48b94d64db4520ad8580c72cbcb8ea16bba486bc315096c2c3ec8cace /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus_qc/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus_qc/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus_qc/execution/docker_cid # return exit code exit $rc [2024-01-10 14:03:01,95] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.stats_n_coverage:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage:/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage:delegated \ us-docker.pkg.dev/general-theiagen/staphb/samtools@sha256:ef92cb5e490c75e66e89ab0a113865839e6954435d363af400eaa27177ff7c91 /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage/execution/docker_cid # return exit code exit $rc [2024-01-10 14:03:01,95] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.stats_n_coverage_primtrim:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage_primtrim/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage_primtrim/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage_primtrim:/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage_primtrim:delegated \ us-docker.pkg.dev/general-theiagen/staphb/samtools@sha256:ef92cb5e490c75e66e89ab0a113865839e6954435d363af400eaa27177ff7c91 /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage_primtrim/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage_primtrim/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-stats_n_coverage_primtrim/execution/docker_cid # return exit code exit $rc [2024-01-10 14:03:02,12] [warn] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.nextclade:NA:1]: Unrecognized runtime attribute keys: disk, dx_instance_type, disks, cpu, memory [2024-01-10 14:03:02,13] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-nextclade/inputs/-1423075165/clearlabs.medaka.consensus.fasta -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/execution/clearlabs.medaka.consensus.fasta: Operation not permitted [2024-01-10 14:03:02,13] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.nextclade:NA:1]: NEXTCLADE_VERSION="$(nextclade --version)" echo $NEXTCLADE_VERSION > NEXTCLADE_VERSION nextclade dataset get --name="sars-cov-2" --reference="MN908947" --tag="2023-09-21T12:00:00Z" -o nextclade_dataset_dir --verbose set -e nextclade run \ --input-dataset=nextclade_dataset_dir/ \ \ \ \ \ \ \ --output-json "clearlabs.medaka.consensus".nextclade.json \ --output-tsv "clearlabs.medaka.consensus".nextclade.tsv \ --output-tree "clearlabs.medaka.consensus".nextclade.auspice.json \ --output-all=. \ "/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-nextclade/inputs/-1423075165/clearlabs.medaka.consensus.fasta" [2024-01-10 14:03:02,15] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.nextclade:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-nextclade/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-nextclade/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-nextclade:/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-nextclade:delegated \ us-docker.pkg.dev/general-theiagen/nextstrain/nextclade@sha256:f4440021c7f854433bd56433024f38199de27e0ef657e11d5fb28aad4f265a48 /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-nextclade/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-nextclade/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-nextclade/execution/docker_cid # return exit code exit $rc [2024-01-10 14:03:06,33] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.stats_n_coverage_primtrim:NA:1]: job id: 46135 [2024-01-10 14:03:06,33] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.pangolin4:NA:1]: job id: 46101 [2024-01-10 14:03:06,33] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.nextclade:NA:1]: job id: 46161 [2024-01-10 14:03:06,34] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.stats_n_coverage:NA:1]: job id: 46132 [2024-01-10 14:03:06,34] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.stats_n_coverage:NA:1]: Status change from - to Done [2024-01-10 14:03:06,34] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.stats_n_coverage_primtrim:NA:1]: Status change from - to Done [2024-01-10 14:03:06,34] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.sc2_gene_coverage:NA:1]: job id: 46114 [2024-01-10 14:03:06,34] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.sc2_gene_coverage:NA:1]: Status change from - to Done [2024-01-10 14:03:06,35] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.consensus_qc:NA:1]: job id: 46120 [2024-01-10 14:03:06,35] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.pangolin4:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:03:06,35] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.nextclade:NA:1]: Status change from - to Done [2024-01-10 14:03:06,35] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.consensus_qc:NA:1]: Status change from - to Done [2024-01-10 14:03:08,29] [info] WorkflowExecutionActor-7958a2eb-d020-46f1-83d0-6873d5a32972 [7958a2eb]: Starting theiacov_clearlabs.nextclade_output_parser, theiacov_clearlabs.vadr [2024-01-10 14:03:11,53] [info] Assigned new job execution tokens to the following groups: 7958a2eb: 2 [2024-01-10 14:03:11,76] [warn] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.vadr:NA:1]: Unrecognized runtime attribute keys: dx_instance_type, cpu, memory [2024-01-10 14:03:11,77] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-vadr/inputs/-1423075165/clearlabs.medaka.consensus.fasta -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-consensus/execution/clearlabs.medaka.consensus.fasta: Operation not permitted [2024-01-10 14:03:11,78] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.vadr:NA:1]: set -e if [ 23493 -gt 10000 ]; then # remove terminal ambiguous nucleotides /opt/vadr/vadr/miniscripts/fasta-trim-terminal-ambigs.pl \ "/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-vadr/inputs/-1423075165/clearlabs.medaka.consensus.fasta" \ --minlen 50 \ --maxlen 30000 \ > "clearlabs.medaka.consensus_trimmed.fasta" # run VADR # --split and --cpu must be used in conjuction v-annotate.pl \ --split --cpu 2 \ --noseqnamemax --glsearch -s -r --nomisc --mkey sarscov2 --lowsim5seq 6 --lowsim3seq 6 --alt_fail lowscore,insertnn,deletinn --out_allfasta \ "clearlabs.medaka.consensus_trimmed.fasta" \ "clearlabs.medaka.consensus" # package everything for output tar -C "clearlabs.medaka.consensus" -czvf "clearlabs.medaka.consensus.vadr.tar.gz" . # package up FASTA files into zip file for output. Note: this will work whether the --out_allfasta flag is included or not (there are just more when the option is used) mkdir -v vadr_fasta_files cp -v clearlabs.medaka.consensus/*.fa vadr_fasta_files zip clearlabs.medaka.consensus_vadr-fasta-files.zip vadr_fasta_files/*.fa # prep alerts into a tsv file for parsing cut -f 5 "clearlabs.medaka.consensus/clearlabs.medaka.consensus.vadr.alt.list" | tail -n +2 > "clearlabs.medaka.consensus.vadr.alerts.tsv" cat "clearlabs.medaka.consensus.vadr.alerts.tsv" | wc -l > NUM_ALERTS else echo "VADR skipped due to poor assembly; assembly length (unambiguous) = 23493" > NUM_ALERTS fi [2024-01-10 14:03:11,80] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.vadr:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-vadr/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-vadr/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-vadr:/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-vadr:delegated \ us-docker.pkg.dev/general-theiagen/staphb/vadr@sha256:f15a73a6f6e0802c3a9c05e7c602740e7029a2e1ccc14aa6935b459be87c98d7 /cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-vadr/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-vadr/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-vadr/execution/docker_cid # return exit code exit $rc [2024-01-10 14:03:11,97] [warn] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.nextclade_output_parser:NA:1]: Unrecognized runtime attribute keys: disk, dx_instance_type, disks, cpu, memory [2024-01-10 14:03:11,98] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-nextclade_output_parser/inputs/-133504526/clearlabs.medaka.consensus.nextclade.tsv -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-nextclade/execution/clearlabs.medaka.consensus.nextclade.tsv: Operation not permitted [2024-01-10 14:03:11,98] [info] BackgroundConfigAsyncJobExecutionActor [7958a2ebtheiacov_clearlabs.nextclade_output_parser:NA:1]: # Set WDL input variable to input.tsv file cat "/cromwell-executions/theiacov_clearlabs/7958a2eb-d020-46f1-83d0-6873d5a32972/call-nextclade_output_parser/inputs/-133504526/clearlabs.medaka.consensus.nextclade.tsv" > input.tsv touch TAMIFLU_AASUBS # Parse outputs using python3 python3 <

✅ theiacov_clearlabs via cromwell also succeeded, exit code was 0 and cromwell reported success

kapsakcj commented 9 months ago

My terminal scrollback buffer didn't catch every single line of output from cromwell but here's the command:

$ cromwell run -i ./tests/inputs/theiacov/wf_theiacov_illumina_pe.json -m metadata.json ./workflows/theiacov/wf_theiacov_illumina_pe.wdl
and here's most of the output. The ending shows the successful workflow message from cromwell. also had a 0 exit code ```bash INFO: Index IX_JOB_STORE_ENTRY_WEU created Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WN created Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WS created Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: Index IX_WORKFLOW_STORE_ENTRY_WS created Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_again.xml::standardize_column_names_again::kshakir ran successfully in 197ms Running Changeset: changesets/rename_workflow_options_in_metadata.xml::rename_workflow_options_in_metadata::tjeandet [2024-01-10 14:06:02,92] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000 [2024-01-10 14:06:02,94] [info] [RenameWorkflowOptionsInMetadata] 100% Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: RenameWorkflowOptionsInMetadata complete. Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: ChangeSet changesets/rename_workflow_options_in_metadata.xml::rename_workflow_options_in_metadata::tjeandet ran successfully in 22ms Running Changeset: changesets/encrypt_and_clear_workflow_options.xml::encrypt_workflow_store_entry_workflow_options::kshakir Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: EncryptWorkflowStoreEntryWorkflowOptions complete. Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: ChangeSet changesets/encrypt_and_clear_workflow_options.xml::encrypt_workflow_store_entry_workflow_options::kshakir ran successfully in 2ms Running Changeset: changesets/encrypt_and_clear_workflow_options.xml::clear_metadata_entry_workflow_options::kshakir Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: ClearMetadataEntryWorkflowOptions complete. Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: ChangeSet changesets/encrypt_and_clear_workflow_options.xml::clear_metadata_entry_workflow_options::kshakir ran successfully in 1ms Running Changeset: changesets/sub_workflow_store.xml::SUB_WORKFLOW_STORE_ENTRY::tjeandet Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: Table SUB_WORKFLOW_STORE_ENTRY created Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::SUB_WORKFLOW_STORE_ENTRY::tjeandet ran successfully in 3ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_uuid_index::tjeandet Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: Index IX_SUB_WORKFLOW_STORE_ENTRY_PWEU created Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_uuid_index::tjeandet ran successfully in 2ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_jobkey_index::tjeandet Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: Unique constraint added to SUB_WORKFLOW_STORE_ENTRY(PARENT_WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, CALL_INDEX, CALL_ATTEMPT) Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_jobkey_index::tjeandet ran successfully in 2ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_root_workflow_fk::tjeandet Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: Foreign key constraint added to SUB_WORKFLOW_STORE_ENTRY (ROOT_WORKFLOW_ID) Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_root_workflow_fk::tjeandet ran successfully in 2ms Running Changeset: changesets/workflow_store_imports_file.xml::workflow-store-imports-file::cjllanwarne Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: Columns IMPORTS_ZIP(BLOB) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_imports_file.xml::workflow-store-imports-file::cjllanwarne ran successfully in 2ms Running Changeset: changesets/workflow_store_labels_file.xml::workflow-store-labels-file::cjllanwarne Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: Columns CUSTOM_LABELS(LONGVARCHAR) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:06:02 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_labels_file.xml::workflow-store-labels-file::cjllanwarne ran successfully in 3ms Running Changeset: changesets/embiggen_detritus_value.xml::embiggen_call_caching_detritus_entry::rmunshi Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE datatype was changed to LONGVARCHAR Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/embiggen_detritus_value.xml::embiggen_call_caching_detritus_entry::rmunshi ran successfully in 2ms Running Changeset: changesets/standardize_column_names_patches.xml::standardize_column_names_patches::kshakir Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_HASH_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key FK_JOB_STORE_RESULT_SIMPLETON_JOB_STORE_ENTRY_ID dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key FK_SUB_WORKFLOW_STORE_ROOT_WORKFLOW_ID_WORKFLOW_STORE_ENTRY_ID dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint UC_CALL_CACHING_ENTRY_WEU_CQFN_JI dropped from CALL_CACHING_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint UC_CALL_CACHING_HASH_ENTRY_CCEI dropped from CALL_CACHING_HASH_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_HASH_ENTRY(CALL_CACHING_ENTRY_ID, HASH_KEY) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key constraint added to SUB_WORKFLOW_STORE_ENTRY (ROOT_WORKFLOW_ID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_patches.xml::standardize_column_names_patches::kshakir ran successfully in 17ms Running Changeset: changesets/standardize_column_names_patches.xml::hsqldb_longvarchar_to_longtext::kshakir Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE datatype was changed to LONGTEXT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.SIMPLETON_VALUE datatype was changed to LONGTEXT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: JOB_STORE_ENTRY.EXCEPTION_MESSAGE datatype was changed to LONGTEXT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: JOB_STORE_SIMPLETON_ENTRY.SIMPLETON_VALUE datatype was changed to LONGTEXT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.CUSTOM_LABELS datatype was changed to LONGTEXT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_DEFINITION datatype was changed to LONGTEXT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_INPUTS datatype was changed to LONGTEXT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_OPTIONS datatype was changed to LONGTEXT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_patches.xml::hsqldb_longvarchar_to_longtext::kshakir ran successfully in 14ms Running Changeset: changesets/nullable_lobs.xml::nullable_lobs::kshakir Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Null constraint dropped from CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Null constraint dropped from CALL_CACHING_SIMPLETON_ENTRY.SIMPLETON_VALUE Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Null constraint dropped from JOB_STORE_SIMPLETON_ENTRY.SIMPLETON_VALUE Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_DEFINITION Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_INPUTS Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_OPTIONS Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.CUSTOM_LABELS Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/nullable_lobs.xml::nullable_lobs::kshakir ran successfully in 25ms Running Changeset: changesets/add_attempt_in_call_caching_entry.xml::add_attempt_in_call_caching_entry::tjeandet Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Columns JOB_ATTEMPT(INT) added to CALL_CACHING_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/add_attempt_in_call_caching_entry.xml::add_attempt_in_call_caching_entry::tjeandet ran successfully in 3ms Running Changeset: changesets/replace_empty_custom_labels.xml::custom_labels_not_null::rmunshi Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_STORE_ENTRY.CUSTOM_LABELS Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/replace_empty_custom_labels.xml::custom_labels_not_null::rmunshi ran successfully in 3ms Running Changeset: changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry::tjeandet Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table CALL_CACHING_AGGREGATION_ENTRY created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry::tjeandet ran successfully in 2ms Running Changeset: changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry_keys_and_indexes::tjeandet Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_AGGREGATION_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index IX_CALL_CACHING_AGGREGATION_ENTRY_BA_IFA created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry_keys_and_indexes::tjeandet ran successfully in 5ms Running Changeset: changesets/custom_label_entry.xml::custom_LABEL_entry::rmunshi Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::custom_LABEL_entry::rmunshi ran successfully in 3ms Running Changeset: changesets/custom_label_entry.xml::WMSE_workflow_execution_uuid_foreign_key::rmunshi Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key constraint added to CUSTOM_LABEL_ENTRY (WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::WMSE_workflow_execution_uuid_foreign_key::rmunshi ran successfully in 3ms Running Changeset: changesets/custom_label_entry.xml::custom_label_entry_index::rmunshi Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, CUSTOM_LABEL_VALUE, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::custom_label_entry_index::rmunshi ran successfully in 2ms Running Changeset: changesets/custom_label_entry.xml::modify label key/value data types::rmunshi Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_KEY datatype was changed to VARCHAR(63) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_VALUE datatype was changed to VARCHAR(63) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::modify label key/value data types::rmunshi ran successfully in 2ms Running Changeset: changesets/docker_hash_store.xml::DOCKER_HASH_STORE_ENTRY::tjeandet Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table DOCKER_HASH_STORE_ENTRY created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store.xml::DOCKER_HASH_STORE_ENTRY::tjeandet ran successfully in 3ms Running Changeset: changesets/docker_hash_store.xml::docker_hash_store_uuid_unique_constraint::tjeandet Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint added to DOCKER_HASH_STORE_ENTRY(WORKFLOW_EXECUTION_UUID, DOCKER_TAG) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store.xml::docker_hash_store_uuid_unique_constraint::tjeandet ran successfully in 2ms Running Changeset: changesets/workflow_store_type_and_version.xml::workflow-store-type-and-version::mcovarr Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Columns WORKFLOW_TYPE(VARCHAR(30)),WORKFLOW_TYPE_VERSION(VARCHAR(255)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_type_and_version.xml::workflow-store-type-and-version::mcovarr ran successfully in 2ms Running Changeset: changesets/remove_pre_pbe_tables.xml::drop-pre-pbe-tables::mcovarr Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table EXECUTION_INFO dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table EXECUTION_EVENT dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table FAILURE_EVENT dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table RUNTIME_ATTRIBUTES dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table SYMBOL dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table EXECUTION dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table WORKFLOW_EXECUTION_AUX dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table WORKFLOW_EXECUTION dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/remove_pre_pbe_tables.xml::drop-pre-pbe-tables::mcovarr ran successfully in 11ms Running Changeset: changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key FK_CUSTOM_LABEL_ENTRY_WORKFLOW_EXECUTION_UUID dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint UC_CUSTOM_LABEL_ENTRY_CLK_CLV_WEU dropped from CUSTOM_LABEL_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint UC_SUMMARY_STATUS_ENTRY_STN_STN dropped from SUMMARY_STATUS_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint UC_WORKFLOW_METADATA_SUMMARY_ENTRY_WEU dropped from WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table METADATA_ENTRY dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table SUMMARY_STATUS_ENTRY dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY_ENTRY dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir ran successfully in 9ms Running Changeset: changesets/workflow_store_state_widening.xml::workflow-store-state-widening::tjeandet Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_STATE datatype was changed to varchar(20) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_state_widening.xml::workflow-store-state-widening::tjeandet ran successfully in 1ms Running Changeset: changesets/workflow_store_restarted_column.xml::workflow-store-restarted-column::tjeandet Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Columns RESTARTED(BOOLEAN) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_restarted_column.xml::workflow-store-restarted-column::tjeandet ran successfully in 3ms Running Changeset: changesets/workflow_store_restarted_column.xml::update-restartable::tjeandet Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Data updated in WORKFLOW_STORE_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Data updated in WORKFLOW_STORE_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_restarted_column.xml::update-restartable::tjeandet ran successfully in 3ms Running Changeset: changesets/workflow_store_workflow_root_column.xml::workflow-store-workflow-root-column::tjeandet Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Columns WORKFLOW_ROOT(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_workflow_root_column.xml::workflow-store-workflow-root-column::tjeandet ran successfully in 2ms Running Changeset: changesets/workflow_store_horizontal_db.xml::workflow-store-horizontal-db::mcovarr Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Column WORKFLOW_STORE_ENTRY.RESTARTED dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Columns CROMWELL_ID(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Columns HEARTBEAT_TIMESTAMP(TIMESTAMP) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_horizontal_db.xml::workflow-store-horizontal-db::mcovarr ran successfully in 5ms Running Changeset: changesets/add_workflow_url_in_workflow_store_entry.xml::add_workflow_url_in_workflow_store_entry::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Columns WORKFLOW_URL(VARCHAR(255)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/add_workflow_url_in_workflow_store_entry.xml::add_workflow_url_in_workflow_store_entry::sshah ran successfully in 2ms Running Changeset: changesets/change_max_size_workflow_url.xml::change_max_size_for_workflow_url::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_URL datatype was changed to VARCHAR(2000) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/change_max_size_workflow_url.xml::change_max_size_for_workflow_url::sshah ran successfully in 2ms Running Changeset: changesets/docker_hash_store_add_size_column.xml::docker_hash_store_entry_add_size_column::tjeandet Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Columns DOCKER_SIZE(BIGINT) added to DOCKER_HASH_STORE_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store_add_size_column.xml::docker_hash_store_entry_add_size_column::tjeandet ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_hash_entry_id.xml::restore_auto_increment_call_caching_hash_entry_id::mcovarr Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_hash_entry_id.xml::restore_auto_increment_call_caching_hash_entry_id::mcovarr ran successfully in 3ms Running Changeset: changesets/add_hog_group_in_workflow_store.xml::add_hog_group_in_workflow_store::cjllanwarne Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Columns HOG_GROUP(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/add_hog_group_in_workflow_store.xml::add_hog_group_in_workflow_store::cjllanwarne ran successfully in 2ms Running Changeset: changesets/resync_engine_schema.xml::resync-engine-schema::kshakir Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.HEARTBEAT_TIMESTAMP datatype was changed to DATETIME(6) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.SUBMISSION_TIME datatype was changed to DATETIME(6) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_STORE_ENTRY.SUBMISSION_TIME Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/resync_engine_schema.xml::resync-engine-schema::kshakir ran successfully in 4ms Running Changeset: changesets/enlarge_job_store_ids.xml::drop_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key FK_JOB_STORE_SIMPLETON_ENTRY_JOB_STORE_ENTRY_ID dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::drop_job_store_simpleton_entry_fk::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_entry_id::mcovarr Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_ENTRY.JOB_STORE_ENTRY_ID Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_entry_id::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_id::mcovarr Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_SIMPLETON_ENTRY_ID Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_id::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_fk::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::recreate_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::recreate_job_store_simpleton_entry_fk::mcovarr ran successfully in 3ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_AGGREGATION_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah ran successfully in 3ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_detritus_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_DETRITUS_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_detritus_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_hash_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_HASH_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_hash_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_SIMPLETON_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah ran successfully in 3ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_cache_entry_id::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CALL_CACHING_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_cache_entry_id::sshah ran successfully in 3ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::reset_call_cache_entry_id_autoincrement::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_ENTRY.CALL_CACHING_ENTRY_ID Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::reset_call_cache_entry_id_autoincrement::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_aggregation_entry_fk::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_aggregation_entry_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_detritus_entry_fk::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_detritus_entry_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_hash_entry_fk::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CALL_CACHING_HASH_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_hash_entry_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_simpleton_entry_fk::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_simpleton_entry_fk::sshah ran successfully in 3ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_AGGREGATION_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah ran successfully in 3ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_detritus_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_DETRITUS_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_detritus_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_hash_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_hash_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_SIMPLETON_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_aggregation_entry_id.xml::enlarge_call_cache_aggregation_entry_id::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_AGGREGATION_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_aggregation_entry_id.xml::enlarge_call_cache_aggregation_entry_id::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_aggregation_entry_id.xml::reset_call_caching_aggregation_entry_id_autoincrement::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_AGGREGATION_ENTRY_ID Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_aggregation_entry_id.xml::reset_call_caching_aggregation_entry_id_autoincrement::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_detritus_entry_id.xml::enlarge_call_cache_detritus_entry_id::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_DETRITUS_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_detritus_entry_id.xml::enlarge_call_cache_detritus_entry_id::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_detritus_entry_id.xml::reset_call_caching_detritus_entry_id_autoincrement::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_DETRITUS_ENTRY_ID Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_detritus_entry_id.xml::reset_call_caching_detritus_entry_id_autoincrement::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_simpleton_entry_id.xml::enlarge_call_cache_simpleton_entry_id::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_SIMPLETON_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_simpleton_entry_id.xml::enlarge_call_cache_simpleton_entry_id::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_simpleton_entry_id.xml::reset_call_caching_simpleton_entry_id_autoincrement::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_SIMPLETON_ENTRY_ID Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_simpleton_entry_id.xml::reset_call_caching_simpleton_entry_id_autoincrement::sshah ran successfully in 2ms Running Changeset: changesets/reset_call_caching_hash_entry_id_autoincrement.xml::reset_call_caching_hash_entry_id_autoincrement::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/reset_call_caching_hash_entry_id_autoincrement.xml::reset_call_caching_hash_entry_id_autoincrement::sshah ran successfully in 2ms Jan 10, 2024 2:06:03 PM liquibase.lockservice INFO: Successfully released change log lock [2024-01-10 14:06:03,37] [info] Running with database db.url = jdbc:hsqldb:mem:591152cf-b38c-46fb-b04e-7995718e277d;shutdown=false;hsqldb.tx=mvcc Jan 10, 2024 2:06:03 PM liquibase.database INFO: Set default schema name to PUBLIC Jan 10, 2024 2:06:03 PM liquibase.lockservice INFO: Successfully acquired change log lock Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Creating database history table with name: PUBLIC.SQLMETADATADATABASECHANGELOG Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Reading from PUBLIC.SQLMETADATADATABASECHANGELOG Running Changeset: metadata_changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table METADATA_ENTRY created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table SUMMARY_STATUS_ENTRY created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY_ENTRY created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, CUSTOM_LABEL_VALUE, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS_ENTRY(SUMMARY_TABLE_NAME, SUMMARIZED_TABLE_NAME) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_METADATA_SUMMARY_ENTRY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Foreign key constraint added to CUSTOM_LABEL_ENTRY (WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WN created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WS created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index METADATA_JOB_AND_KEY_IDX created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index METADATA_JOB_IDX created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index METADATA_WORKFLOW_IDX created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index SYS_IDX_11226 created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir ran successfully in 23ms Running Changeset: changesets/change_max_size_label_entry.xml::change_max_size_for_key_and_value_in_label::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_KEY datatype was changed to VARCHAR(255) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_VALUE datatype was changed to VARCHAR(255) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet changesets/change_max_size_label_entry.xml::change_max_size_for_key_and_value_in_label::sshah ran successfully in 2ms Running Changeset: metadata_changesets/delete_duplicate_custom_labels.xml::delete_duplicate_custom_labels::kshakir Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Data deleted from CUSTOM_LABEL_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint UC_CUSTOM_LABEL_ENTRY_CLK_CLV_WEU dropped from CUSTOM_LABEL_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/delete_duplicate_custom_labels.xml::delete_duplicate_custom_labels::kshakir ran successfully in 5ms Running Changeset: metadata_changesets/add_submission_timestamp_metadata_summary.xml::add_submission_timestamp_in_metadata_entry::sshah Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Columns SUBMISSION_TIMESTAMP(DATETIME(6)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_submission_timestamp_metadata_summary.xml::add_submission_timestamp_in_metadata_entry::sshah ran successfully in 2ms Running Changeset: metadata_changesets/custom_label_entry_key_value_index.xml::custom_label_entry_key_value_index::mcovarr Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index IX_CUSTOM_LABEL_ENTRY_CLK_CLV created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/custom_label_entry_key_value_index.xml::custom_label_entry_key_value_index::mcovarr ran successfully in 2ms Running Changeset: metadata_changesets/add_parent_and_root_workflow_execution_uuid.xml::add_parent_and_root_workflow_execution_uuid::kshakir Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Columns PARENT_WORKFLOW_EXECUTION_UUID(VARCHAR(100)),ROOT_WORKFLOW_EXECUTION_UUID(VARCHAR(100)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_PWEU created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_RWEU created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint UC_SUMMARY_STATUS_ENTRY_STN_STN dropped from SUMMARY_STATUS_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.SUMMARY_TABLE_NAME renamed to SUMMARY_NAME Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.SUMMARIZED_TABLE_NAME dropped Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.MAXIMUM_ID renamed to SUMMARY_POSITION Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS_ENTRY(SUMMARY_NAME) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_parent_and_root_workflow_execution_uuid.xml::add_parent_and_root_workflow_execution_uuid::kshakir ran successfully in 11ms Running Changeset: metadata_changesets/add_parent_and_root_workflow_typo_fix.xml::add_parent_and_root_workflow_typo_fix::kshakir Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Data deleted from SUMMARY_STATUS_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_parent_and_root_workflow_typo_fix.xml::add_parent_and_root_workflow_typo_fix::kshakir ran successfully in 4ms Running Changeset: metadata_changesets/metadata_index_removals.xml::metadata_index_removals::mcovarr Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index METADATA_JOB_AND_KEY_IDX dropped from table METADATA_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index METADATA_JOB_IDX dropped from table METADATA_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/metadata_index_removals.xml::metadata_index_removals::mcovarr ran successfully in 4ms Running Changeset: metadata_changesets/add_metadata_archive_status.xml::add_metadata_archive_status::cjllanwarne Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Columns METADATA_ARCHIVE_STATUS(VARCHAR(30)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_metadata_archive_status.xml::add_metadata_archive_status::cjllanwarne ran successfully in 3ms Running Changeset: metadata_changesets/add_metadata_archive_status.xml::metadata_archive_status_index::cjllanwarne Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_metadata_archive_status.xml::metadata_archive_status_index::cjllanwarne ran successfully in 2ms Running Changeset: metadata_changesets/summarization_queue_table.xml::summary_queue_table::gsterin Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Table SUMMARY_QUEUE_ENTRY created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table.xml::summary_queue_table::gsterin ran successfully in 1ms Running Changeset: metadata_changesets/summarization_queue_table.xml::summary_queue_initialization::gsterin Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table.xml::summary_queue_initialization::gsterin ran successfully in 4ms Running Changeset: metadata_changesets/summarization_queue_table_add_primary_key.xml::summary_queue_add_primary_key::gsterin Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Primary key added to SUMMARY_QUEUE_ENTRY (METADATA_JOURNAL_ID) Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table_add_primary_key.xml::summary_queue_add_primary_key::gsterin ran successfully in 2ms Running Changeset: metadata_changesets/remove_non_summarizable_metadata_from_queue.xml::delete_non_summarizable_metadata_from_queue::mcovarr Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/remove_non_summarizable_metadata_from_queue.xml::delete_non_summarizable_metadata_from_queue::mcovarr ran successfully in 3ms Running Changeset: metadata_changesets/update_metadata_archive_index.xml::update_old_metadata_archive_status_index::cjllanwarne Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS dropped from table WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS_ET created Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/update_metadata_archive_index.xml::update_old_metadata_archive_status_index::cjllanwarne ran successfully in 2ms Running Changeset: metadata_changesets/reset_archive_statuses_to_null.xml::reset_archive_statuses_to_null::cjllanwarne Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: Data updated in WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:06:03 PM liquibase.changelog INFO: ChangeSet metadata_changesets/reset_archive_statuses_to_null.xml::reset_archive_statuses_to_null::cjllanwarne ran successfully in 1ms Jan 10, 2024 2:06:03 PM liquibase.lockservice INFO: Successfully released change log lock [2024-01-10 14:06:04,01] [info] Slf4jLogger started [2024-01-10 14:06:04,24] [info] Workflow heartbeat configuration: { "cromwellId" : "cromid-dedd598", "heartbeatInterval" : "2 minutes", "ttl" : "10 minutes", "failureShutdownDuration" : "5 minutes", "writeBatchSize" : 10000, "writeThreshold" : 10000 } [2024-01-10 14:06:04,32] [info] Metadata summary refreshing every 1 second. [2024-01-10 14:06:04,32] [info] No metadata archiver defined in config [2024-01-10 14:06:04,33] [info] No metadata deleter defined in config [2024-01-10 14:06:04,36] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds. [2024-01-10 14:06:04,36] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds. [2024-01-10 14:06:04,38] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds. [2024-01-10 14:06:04,48] [info] JobRestartCheckTokenDispenser - Distribution rate: 50 per 1 seconds. [2024-01-10 14:06:04,53] [info] JobExecutionTokenDispenser - Distribution rate: 20 per 10 seconds. [2024-01-10 14:06:04,62] [info] SingleWorkflowRunnerActor: Version 83 [2024-01-10 14:06:04,63] [info] SingleWorkflowRunnerActor: Submitting workflow [2024-01-10 14:06:04,70] [info] Unspecified type (Unspecified version) workflow f226a562-09f6-40c5-8a8e-81c591f0be48 submitted [2024-01-10 14:06:04,73] [info] SingleWorkflowRunnerActor: Workflow submitted f226a562-09f6-40c5-8a8e-81c591f0be48 [2024-01-10 14:06:04,74] [info] 1 new workflows fetched by cromid-dedd598: f226a562-09f6-40c5-8a8e-81c591f0be48 [2024-01-10 14:06:04,76] [info] WorkflowManagerActor: Starting workflow f226a562-09f6-40c5-8a8e-81c591f0be48 [2024-01-10 14:06:04,77] [info] WorkflowManagerActor: Successfully started WorkflowActor-f226a562-09f6-40c5-8a8e-81c591f0be48 [2024-01-10 14:06:04,77] [info] Retrieved 1 workflows from the WorkflowStoreActor [2024-01-10 14:06:04,80] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes. [2024-01-10 14:06:04,91] [info] MaterializeWorkflowDescriptorActor [f226a562]: Parsing workflow as WDL 1.0 [2024-01-10 14:06:07,23] [info] MaterializeWorkflowDescriptorActor [f226a562]: Call-to-Backend assignments: theiacov_illumina_pe.consensus_qc -> Local, ivar_consensus.consensus -> Local, theiacov_illumina_pe.sc2_gene_coverage -> Local, read_QC_trim_pe.fastq_scan_raw -> Local, ivar_consensus.bwa -> Local, flu_antiviral_substitutions.aa_subs_pb2 -> Local, flu_antiviral_substitutions.flu_antiviral_parser_h3_ha -> Local, flu_antiviral_substitutions.aa_subs_n2_na -> Local, flu_antiviral_substitutions.flu_antiviral_parser_pb1 -> Local, flu_antiviral_substitutions.mafft_n1_na -> Local, read_QC_trim_pe.trimmomatic_pe -> Local, theiacov_illumina_pe.quasitools_illumina_pe -> Local, flu_antiviral_substitutions.mafft_pa -> Local, flu_antiviral_substitutions.flu_antiviral_parser_pb2 -> Local, flu_antiviral_substitutions.aa_subs_h1_ha -> Local, flu_antiviral_substitutions.aa_subs_pa -> Local, ivar_consensus.stats_n_coverage_primtrim -> Local, read_QC_trim_pe.fastq_scan_clean -> Local, theiacov_illumina_pe.nextclade_output_parser_flu_na -> Local, theiacov_illumina_pe.clean_check_reads -> Local, theiacov_illumina_pe.nextclade -> Local, flu_antiviral_substitutions.aa_subs_n1_na -> Local, flu_antiviral_substitutions.flu_antiviral_parser_n1_na -> Local, read_QC_trim_pe.kraken2_standalone -> Local, theiacov_illumina_pe.vadr -> Local, read_QC_trim_pe.bbduk -> Local, read_QC_trim_pe.fastp -> Local, ivar_consensus.variant_call -> Local, flu_antiviral_substitutions.mafft_h3_ha -> Local, read_QC_trim_pe.kraken2_theiacov_dehosted -> Local, flu_antiviral_substitutions.mafft_pb2 -> Local, theiacov_illumina_pe.abricate_flu -> Local, flu_antiviral_substitutions.aa_subs_h3_ha -> Local, read_QC_trim_pe.fastqc_raw -> Local, flu_antiviral_substitutions.flu_antiviral_parser_h1_ha -> Local, flu_antiviral_substitutions.mafft_h1_ha -> Local, ivar_consensus.primer_trim -> Local, read_QC_trim_pe.readlength -> Local, theiacov_illumina_pe.pangolin4 -> Local, read_QC_trim_pe.fastqc_clean -> Local, flu_antiviral_substitutions.flu_antiviral_parser_pa -> Local, ivar_consensus.stats_n_coverage -> Local, theiacov_illumina_pe.qc_check_task -> Local, theiacov_illumina_pe.raw_check_reads -> Local, theiacov_illumina_pe.nextclade_flu_na -> Local, flu_antiviral_substitutions.serialization -> Local, theiacov_illumina_pe.version_capture -> Local, flu_antiviral_substitutions.mafft_n2_na -> Local, flu_antiviral_substitutions.aa_subs_pb1 -> Local, flu_antiviral_substitutions.flu_antiviral_parser_n2_na -> Local, read_QC_trim_pe.kraken2_theiacov_raw -> Local, theiacov_illumina_pe.nextclade_output_parser -> Local, flu_antiviral_substitutions.mafft_pb1 -> Local, theiacov_illumina_pe.irma -> Local, read_QC_trim_pe.midas -> Local, read_QC_trim_pe.ncbi_scrub_pe -> Local [2024-01-10 14:06:07,46] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,46] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,46] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,46] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,46] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,46] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [dx_instance_type, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,47] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:07,48] [warn] Local [f226a562]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:06:08,58] [info] WorkflowExecutionActor-f226a562-09f6-40c5-8a8e-81c591f0be48 [f226a562]: Starting theiacov_illumina_pe.version_capture [2024-01-10 14:06:09,50] [info] Not triggering log of restart checking token queue status. Effective log interval = None [2024-01-10 14:06:09,55] [info] Not triggering log of execution token queue status. Effective log interval = None [2024-01-10 14:06:09,65] [info] WorkflowExecutionActor-f226a562-09f6-40c5-8a8e-81c591f0be48 [f226a562]: Starting theiacov_illumina_pe.raw_check_reads [2024-01-10 14:06:14,56] [info] Assigned new job execution tokens to the following groups: f226a562: 2 [2024-01-10 14:06:15,62] [warn] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.raw_check_reads:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:06:15,62] [warn] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.version_capture:NA:1]: Unrecognized runtime attribute keys: dx_instance_type, disks, cpu, memory [2024-01-10 14:06:15,72] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.version_capture:NA:1]: PHB_Version="PHB v1.2.1" date +"%Y-%m-%d" > TODAY echo "$PHB_Version" > PHB_VERSION [2024-01-10 14:06:15,72] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.raw_check_reads:NA:1]: flag="PASS" # initalize estimated genome size estimated_genome_size=0 if [[ "false" == "false" ]] ; then # set cat command based on compression if [[ "/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads/inputs/1422045004/SRR13687078_1.fastq.gz" == *".gz" ]] ; then cat_reads="zcat" else cat_reads="cat" fi # check one: number of reads read1_num=eval "$cat_reads /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads/inputs/1422045004/SRR13687078_1.fastq.gz" | awk '{s++}END{print s/4}' read2_num=eval "$cat_reads /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads/inputs/1422045004/SRR13687078_2.fastq.gz" | awk '{s++}END{print s/4}' # awk '{s++}END{print s/4' counts the number of lines and divides them by 4 # key assumption: in fastq there will be four lines per read # sometimes fastqs do not have 4 lines per read, so this might fail one day reads_total=$(expr $read1_num + $read2_num) if [ "${reads_total}" -le "113" ]; then flag="FAIL; the total number of reads is below the minimum of 113" else flag="PASS" fi # checks two and three: number of basepairs and proportion of sequence if [ "${flag}" == "PASS" ]; then # count number of basepairs # this only works if the fastq has 4 lines per read, so this might fail one day read1_bp=eval "${cat_reads} /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads/inputs/1422045004/SRR13687078_1.fastq.gz" | paste - - - - | cut -f2 | tr -d '\n' | wc -c read2_bp=eval "${cat_reads} /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads/inputs/1422045004/SRR13687078_2.fastq.gz" | paste - - - - | cut -f2 | tr -d '\n' | wc -c # paste - - - - (print 4 consecutive lines in one row, tab delimited) # cut -f2 print only the second column (the second line of the fastq 4-line) # tr -d '\n' removes line endings # wc -c counts characters # set proportion variables for easy comparison # removing the , 2) to make these integers instead of floats percent_read1=$(python3 -c "print(round(($read1_bp / ($read1_bp + $read2_bp))*100))") percent_read2=$(python3 -c "print(round(($read2_bp / ($read1_bp + $read2_bp))*100))") if [ "$percent_read1" -lt "40" ] ; then flag="FAIL; more than 40 percent of the total sequence is found in R2 (BP: $read2_bp; PERCENT: $percent_read2) compared to R1 (BP: $read1_bp; PERCENT: $percent_read1)" elif [ "$percent_read2" -lt "40" ] ; then flag="FAIL; more than 40 percent of the total sequence is found in R1 (BP: $read1_bp; PERCENT: $percent_read1) compared to R2 (BP: $read2_bp; PERCENT: $percent_read2)" else flag="PASS" fi # check total number of basepairs if [ "$flag" == "PASS" ] ; then bp_total=$(expr $read1_bp + $read2_bp) if [ "${bp_total}" -le "34000" ]; then flag="FAIL; the total number of basepairs is below the minimum of 34000" else flag="PASS" fi fi fi #checks four and five: estimated genome size and coverage if [ "${flag}" == "PASS" ]; then # estimate genome size if theiaprok AND expected_genome_size was not provided if [ "theiacov" == "theiaprok" ] && [[ -z "" ]]; then # First Pass; assuming average depth mash sketch -o test -k 31 -m 3 -r /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads/inputs/1422045004/SRR13687078_1.fastq.gz /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads/inputs/1422045004/SRR13687078_2.fastq.gz > mash-output.txt 2>&1 grep "Estimated genome size:" mash-output.txt | \ awk '{if($4){printf("%5.0f\n", $4)}} END {if (!NR) print "0"}' > genome_size_output grep "Estimated coverage:" mash-output.txt | \ awk '{if($3){printf("%d", $3)}} END {if (!NR) print "0"}' > coverage_output rm -rf test.msh rm -rf mash-output.txt estimated_genome_size=head -n1 genome_size_output estimated_coverage=head -n1 coverage_output # Check if second pass is needed if [ ${estimated_genome_size} -gt "2673870" ] || [ ${estimated_genome_size} -lt "1700" ] ; then # Probably high coverage, try increasing number of kmer copies to 10 M="-m 10" if [ ${estimated_genome_size} -lt "1700" ]; then # Probably low coverage, try decreasing the number of kmer copies to 1 M="-m 1" fi mash sketch -o test -k 31 ${M} -r /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads/inputs/1422045004/SRR13687078_1.fastq.gz /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads/inputs/1422045004/SRR13687078_2.fastq.gz > mash-output.txt 2>&1 grep "Estimated genome size:" mash-output.txt | \ awk '{if($4){printf("%5.0f\n", $4)}} END {if (!NR) print "0"}' > genome_size_output grep "Estimated coverage:" mash-output.txt | \ awk '{if($3){printf("%d", $3)}} END {if (!NR) print "0"}' > coverage_output rm -rf test.msh rm -rf mash-output.txt fi estimated_genome_size=head -n1 genome_size_output estimated_coverage=head -n1 coverage_output # estimate coverage if theiacov OR expected_genome_size was provided elif [ "theiacov" == "theiacov" ] || [ "" ]; then if [ "" ]; then estimated_genome_size= # use user-provided expected_genome_size elif [ "sars-cov-2" == "sars-cov-2" ]; then estimated_genome_size=29903 # size taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_045512.2 elif [ "sars-cov-2" == "MPXV" ]; then estimated_genome_size=197209 # size of 2022 virus taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_063383.1 elif [ "sars-cov-2" == "flu" ]; then estimated_genome_size=14000 # 500 bp over the CDC's approximate full genome size of 13500 (see https://www.cdc.gov/flu/about/professionals/genetic-characterization.htm) elif [ "sars-cov-2" == "HIV" ]; then estimated_genome_size=9181 # size taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_001802.1 elif [ "sars-cov-2" == "WNV" ]; then estimated_genome_size=11092 # WNV lineage 1 size from https://www.ncbi.nlm.nih.gov/nuccore/NC_009942.1 else flag="FAIL; the organism tag provided (sars-cov-2) is not valid and no expected_genome_size was provided." fi # coverage is calculated here by N/G where N is number of bases, and G is genome length # this will nearly always be an overestimation if [ $estimated_genome_size -ne 0 ]; then # prevent divided by zero errors estimated_coverage=$(python3 -c "print(round(($read1_bp+$read2_bp)/$estimated_genome_size))") else # they provided 0 for estimated_genome_size, nice estimated_coverage=0 fi else # workflow series was not provided or no est genome size was provided; default to fail estimated_genome_size=0 estimated_coverage=0 fi if [ "${estimated_genome_size}" -ge "2673870" ] && [ "theiacov" == "theiaprok" ] ; then flag="FAIL; the estimated genome size (${estimated_genome_size}) is larger than the maximum of 2673870 bps" elif [ "${estimated_genome_size}" -le "1700" ] && [ "theiacov" == "theiaprok" ] ; then flag="FAIL; the estimated genome size (${estimated_genome_size}) is smaller than the minimum of 1700 bps" else flag="PASS" if [ "${estimated_coverage}" -lt "10" ] ; then flag="FAIL; the estimated coverage (${estimated_coverage}) is less than the minimum of 10x" else flag="PASS" echo $estimated_genome_size | tee EST_GENOME_LENGTH fi fi fi fi echo $flag | tee FLAG echo $estimated_genome_size | tee EST_GENOME_LENGTH [2024-01-10 14:06:15,87] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.version_capture:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-version_capture/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-version_capture/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-version_capture:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-version_capture:delegated \ us-docker.pkg.dev/general-theiagen/ubuntu/ubuntu@sha256:9a35cef02b57290a7c919824b632897fbe6db3aa18c930110b04d81fcad458ff /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-version_capture/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-version_capture/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-version_capture/execution/docker_cid # return exit code exit $rc [2024-01-10 14:06:15,87] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.raw_check_reads:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads:delegated \ us-docker.pkg.dev/general-theiagen/bactopia/gather_samples@sha256:2e044c7be68519b9d683cd242177518e6db662b40b3bf669f891954f90a4d9a5 /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-raw_check_reads/execution/docker_cid # return exit code exit $rc [2024-01-10 14:06:19,41] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.version_capture:NA:1]: job id: 49235 [2024-01-10 14:06:19,41] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.raw_check_reads:NA:1]: job id: 49237 [2024-01-10 14:06:19,42] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.version_capture:NA:1]: Status change from - to Done [2024-01-10 14:06:19,42] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.raw_check_reads:NA:1]: Status change from - to Done [2024-01-10 14:06:27,02] [info] cbcb01fe-5890-47ca-8ff3-c47190c0201c-SubWorkflowActor-SubWorkflow-read_QC_trim:-1:1 [cbcb01fe]: Starting read_QC_trim_pe.fastq_scan_raw, read_QC_trim_pe.kraken2_theiacov_raw, read_QC_trim_pe.ncbi_scrub_pe [2024-01-10 14:06:34,55] [info] Assigned new job execution tokens to the following groups: f226a562: 3 [2024-01-10 14:06:34,76] [warn] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.kraken2_theiacov_raw:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:06:34,76] [warn] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.ncbi_scrub_pe:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:06:34,77] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.ncbi_scrub_pe:NA:1]: # date and version control date | tee DATE # unzip fwd file as scrub tool does not take in .gz fastq files if [[ "/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/inputs/1422045004/SRR13687078_1.fastq.gz" == *.gz ]] then gunzip -c /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/inputs/1422045004/SRR13687078_1.fastq.gz > r1.fastq read1_unzip=r1.fastq else read1_unzip=/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/inputs/1422045004/SRR13687078_1.fastq.gz fi # dehost reads /opt/scrubber/scripts/scrub.sh -n ${read1_unzip} |& tail -n1 | awk -F" " '{print $1}' > FWD_SPOTS_REMOVED # gzip dehosted reads gzip ${read1_unzip}.clean -c > SRR13687078_R1_dehosted.fastq.gz # do the same on read # unzip file if necessary if [[ "/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/inputs/1422045004/SRR13687078_2.fastq.gz" == *.gz ]] then gunzip -c /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/inputs/1422045004/SRR13687078_2.fastq.gz > r2.fastq read2_unzip=r2.fastq else read2_unzip=/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/inputs/1422045004/SRR13687078_2.fastq.gz fi # dehost reads /opt/scrubber/scripts/scrub.sh -n ${read2_unzip} |& tail -n1 | awk -F" " '{print $1}' > REV_SPOTS_REMOVED # gzip dehosted reads gzip ${read2_unzip}.clean -c > SRR13687078_R2_dehosted.fastq.gz [2024-01-10 14:06:34,78] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.kraken2_theiacov_raw:NA:1]: # date and version control date | tee DATE kraken2 --version | head -n1 | tee VERSION num_reads=$(ls *fastq.gz 2> /dev/nul | wc -l) if ! [ -z /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_raw/inputs/1422045004/SRR13687078_2.fastq.gz ]; then mode="--paired" fi echo $mode kraken2 $mode \ --threads 4 \ --db /kraken2-db \ /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_raw/inputs/1422045004/SRR13687078_1.fastq.gz /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_raw/inputs/1422045004/SRR13687078_2.fastq.gz \ --report SRR13687078_kraken2_report.txt >/dev/null percentage_human=$(grep "Homo sapiens" SRR13687078_kraken2_report.txt | cut -f 1) # | tee PERCENT_HUMAN percentage_sc2=$(grep "Severe acute respiratory syndrome coronavirus 2" SRR13687078_kraken2_report.txt | cut -f1 ) # | tee PERCENT_COV if [ -z "$percentage_human" ] ; then percentage_human="0" ; fi if [ -z "$percentage_sc2" ] ; then percentage_sc2="0" ; fi echo $percentage_human | tee PERCENT_HUMAN echo $percentage_sc2 | tee PERCENT_SC2 # capture target org percentage if [ ! -z "" ]; then echo "Target org designated: " percent_target_org=$(grep "" SRR13687078_kraken2_report.txt | cut -f1 | head -n1 ) if [-z "$percent_target_org" ] ; then percent_target_org="0" ; fi else percent_target_org="" fi echo $percent_target_org | tee PERCENT_TARGET_ORG [2024-01-10 14:06:34,80] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.ncbi_scrub_pe:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe:delegated \ us-docker.pkg.dev/general-theiagen/ncbi/sra-human-scrubber@sha256:b7dba71079344daea4ea3363e1a67fa54edb7ec65459d039669c68a66d38b140 /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/execution/docker_cid # return exit code exit $rc [2024-01-10 14:06:34,81] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.kraken2_theiacov_raw:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_raw/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_raw/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_raw:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_raw:delegated \ us-docker.pkg.dev/general-theiagen/staphb/kraken2@sha256:7884b5a6ec3c20cd7edbd42b92ae7e9806b4581997acb270b1b9ecb627474e2a /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_raw/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_raw/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_raw/execution/docker_cid # return exit code exit $rc [2024-01-10 14:06:34,87] [warn] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.fastq_scan_raw:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:06:34,88] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.fastq_scan_raw:NA:1]: # capture date and version date | tee DATE fastq-scan -v | tee VERSION # set cat command based on compression if [[ "/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_raw/inputs/1422045004/SRR13687078_1.fastq.gz" == *".gz" ]] ; then cat_reads="zcat" else cat_reads="cat" fi # capture forward read stats eval "${cat_reads} /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_raw/inputs/1422045004/SRR13687078_1.fastq.gz" | fastq-scan | tee SRR13687078_1_fastq-scan.json cat SRR13687078_1_fastq-scan.json | jq .qc_stats.read_total | tee READ1_SEQS read1_seqs=$(cat READ1_SEQS) eval "${cat_reads} /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_raw/inputs/1422045004/SRR13687078_2.fastq.gz" | fastq-scan | tee SRR13687078_2_fastq-scan.json cat SRR13687078_2_fastq-scan.json | jq .qc_stats.read_total | tee READ2_SEQS read2_seqs=$(cat READ2_SEQS) # capture number of read pairs if [ "${read1_seqs}" == "${read2_seqs}" ]; then read_pairs=${read1_seqs} else read_pairs="Uneven pairs: R1=${read1_seqs}, R2=${read2_seqs}" fi echo $read_pairs | tee READ_PAIRS [2024-01-10 14:06:34,92] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.fastq_scan_raw:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_raw/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_raw/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_raw:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_raw:delegated \ us-docker.pkg.dev/general-theiagen/biocontainers/fastq-scan@sha256:dd4880ca592ef724a15cd63dfcb4e3e04f5d4b269976e563ab7413d8ca5cfcfa /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_raw/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_raw/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_raw/execution/docker_cid # return exit code exit $rc [2024-01-10 14:06:39,39] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.ncbi_scrub_pe:NA:1]: job id: 49655 [2024-01-10 14:06:39,39] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.kraken2_theiacov_raw:NA:1]: job id: 49665 [2024-01-10 14:06:39,39] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.fastq_scan_raw:NA:1]: job id: 49681 [2024-01-10 14:06:39,39] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.fastq_scan_raw:NA:1]: Status change from - to Done [2024-01-10 14:06:39,39] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.ncbi_scrub_pe:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:06:39,39] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.kraken2_theiacov_raw:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:06:47,01] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.kraken2_theiacov_raw:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:06:52,69] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.ncbi_scrub_pe:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:06:55,51] [info] cbcb01fe-5890-47ca-8ff3-c47190c0201c-SubWorkflowActor-SubWorkflow-read_QC_trim:-1:1 [cbcb01fe]: Starting read_QC_trim_pe.trimmomatic_pe, read_QC_trim_pe.kraken2_theiacov_dehosted [2024-01-10 14:07:04,54] [info] Assigned new job execution tokens to the following groups: f226a562: 2 [2024-01-10 14:07:04,55] [warn] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.kraken2_theiacov_dehosted:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:07:04,56] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_dehosted/inputs/1824272822/SRR13687078_R1_dehosted.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/execution/SRR13687078_R1_dehosted.fastq.gz: Operation not permitted [2024-01-10 14:07:04,57] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_dehosted/inputs/1824272822/SRR13687078_R2_dehosted.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/execution/SRR13687078_R2_dehosted.fastq.gz: Operation not permitted [2024-01-10 14:07:04,57] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.kraken2_theiacov_dehosted:NA:1]: # date and version control date | tee DATE kraken2 --version | head -n1 | tee VERSION num_reads=$(ls *fastq.gz 2> /dev/nul | wc -l) if ! [ -z /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_dehosted/inputs/1824272822/SRR13687078_R2_dehosted.fastq.gz ]; then mode="--paired" fi echo $mode kraken2 $mode \ --threads 4 \ --db /kraken2-db \ /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_dehosted/inputs/1824272822/SRR13687078_R1_dehosted.fastq.gz /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_dehosted/inputs/1824272822/SRR13687078_R2_dehosted.fastq.gz \ --report SRR13687078_kraken2_report.txt >/dev/null percentage_human=$(grep "Homo sapiens" SRR13687078_kraken2_report.txt | cut -f 1) # | tee PERCENT_HUMAN percentage_sc2=$(grep "Severe acute respiratory syndrome coronavirus 2" SRR13687078_kraken2_report.txt | cut -f1 ) # | tee PERCENT_COV if [ -z "$percentage_human" ] ; then percentage_human="0" ; fi if [ -z "$percentage_sc2" ] ; then percentage_sc2="0" ; fi echo $percentage_human | tee PERCENT_HUMAN echo $percentage_sc2 | tee PERCENT_SC2 # capture target org percentage if [ ! -z "" ]; then echo "Target org designated: " percent_target_org=$(grep "" SRR13687078_kraken2_report.txt | cut -f1 | head -n1 ) if [-z "$percent_target_org" ] ; then percent_target_org="0" ; fi else percent_target_org="" fi echo $percent_target_org | tee PERCENT_TARGET_ORG [2024-01-10 14:07:04,60] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.kraken2_theiacov_dehosted:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_dehosted/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_dehosted/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_dehosted:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_dehosted:delegated \ us-docker.pkg.dev/general-theiagen/staphb/kraken2@sha256:7884b5a6ec3c20cd7edbd42b92ae7e9806b4581997acb270b1b9ecb627474e2a /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_dehosted/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_dehosted/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_dehosted/execution/docker_cid # return exit code exit $rc [2024-01-10 14:07:04,73] [warn] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.trimmomatic_pe:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:07:04,74] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-trimmomatic_pe/inputs/1824272822/SRR13687078_R1_dehosted.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/execution/SRR13687078_R1_dehosted.fastq.gz: Operation not permitted [2024-01-10 14:07:04,75] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-trimmomatic_pe/inputs/1824272822/SRR13687078_R2_dehosted.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/execution/SRR13687078_R2_dehosted.fastq.gz: Operation not permitted [2024-01-10 14:07:04,75] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.trimmomatic_pe:NA:1]: # date and version control date | tee DATE trimmomatic -version > VERSION && sed -i -e 's/^/Trimmomatic /' VERSION trimmomatic PE \ \ -threads 4 \ /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-trimmomatic_pe/inputs/1824272822/SRR13687078_R1_dehosted.fastq.gz /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-trimmomatic_pe/inputs/1824272822/SRR13687078_R2_dehosted.fastq.gz \ -baseout SRR13687078.fastq.gz \ SLIDINGWINDOW:4:30 \ MINLEN:75 &> SRR13687078.trim.stats.txt [2024-01-10 14:07:04,78] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.trimmomatic_pe:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-trimmomatic_pe/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-trimmomatic_pe/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-trimmomatic_pe:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-trimmomatic_pe:delegated \ us-docker.pkg.dev/general-theiagen/staphb/trimmomatic@sha256:57b673e66313e355a447e4fa1a78fd3ba1ae3ddd8c8f91358efe99140acb5ddb /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-trimmomatic_pe/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-trimmomatic_pe/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-trimmomatic_pe/execution/docker_cid # return exit code exit $rc [2024-01-10 14:07:09,38] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.trimmomatic_pe:NA:1]: job id: 50355 [2024-01-10 14:07:09,38] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.kraken2_theiacov_dehosted:NA:1]: job id: 50330 [2024-01-10 14:07:09,38] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.trimmomatic_pe:NA:1]: Status change from - to Done [2024-01-10 14:07:09,38] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.kraken2_theiacov_dehosted:NA:1]: Status change from - to Done [2024-01-10 14:07:11,81] [info] cbcb01fe-5890-47ca-8ff3-c47190c0201c-SubWorkflowActor-SubWorkflow-read_QC_trim:-1:1 [cbcb01fe]: Starting read_QC_trim_pe.bbduk [2024-01-10 14:07:14,54] [info] Assigned new job execution tokens to the following groups: f226a562: 1 [2024-01-10 14:07:14,71] [warn] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.bbduk:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:07:14,71] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/inputs/-1950311358/SRR13687078_2P.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-trimmomatic_pe/execution/SRR13687078_2P.fastq.gz: Operation not permitted [2024-01-10 14:07:14,72] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/inputs/-1950311358/SRR13687078_1P.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-trimmomatic_pe/execution/SRR13687078_1P.fastq.gz: Operation not permitted [2024-01-10 14:07:14,72] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.bbduk:NA:1]: # date and version control date | tee DATE # set adapter fasta if [[ ! -z "" ]]; then echo "Using user supplied FASTA file for adapters..." adapter_fasta="" else echo "User did not supply adapters FASTA file, using default adapters.fa file..." adapter_fasta="/bbmap/resources/adapters.fa" fi # set phix fasta if [[ ! -z "" ]]; then echo "Using user supplied FASTA file for phiX..." phix_fasta="" else echo "User did not supply phiX FASTA file, using default phix174_ill.ref.fa.gz file..." phix_fasta="/bbmap/resources/phix174_ill.ref.fa.gz" fi repair.sh in1=/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/inputs/-1950311358/SRR13687078_1P.fastq.gz in2=/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/inputs/-1950311358/SRR13687078_2P.fastq.gz out1=SRR13687078.paired_1.fastq.gz out2=SRR13687078.paired_2.fastq.gz bbduk.sh in1=SRR13687078.paired_1.fastq.gz in2=SRR13687078.paired_2.fastq.gz out1=SRR13687078.rmadpt_1.fastq.gz out2=SRR13687078.rmadpt_2.fastq.gz ref=${adapter_fasta} stats=SRR13687078.adapters.stats.txt ktrim=r k=23 mink=11 hdist=1 tpe tbo ordered=t bbduk.sh in1=SRR13687078.rmadpt_1.fastq.gz in2=SRR13687078.rmadpt_2.fastq.gz out1=SRR13687078_1.clean.fastq.gz out2=SRR13687078_2.clean.fastq.gz outm=SRR13687078.matched_phix.fq ref=${phix_fasta} k=31 hdist=1 stats=SRR13687078.phix.stats.txt ordered=t [2024-01-10 14:07:14,73] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.bbduk:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk:delegated \ us-docker.pkg.dev/general-theiagen/staphb/bbtools@sha256:fe632b5aa277b939efcf9cdfdd65703133ba3c01ec50db2833b2bb3cf7048ec2 /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/execution/docker_cid # return exit code exit $rc [2024-01-10 14:07:19,37] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.bbduk:NA:1]: job id: 50774 [2024-01-10 14:07:19,37] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.bbduk:NA:1]: Status change from - to Done [2024-01-10 14:07:20,96] [info] cbcb01fe-5890-47ca-8ff3-c47190c0201c-SubWorkflowActor-SubWorkflow-read_QC_trim:-1:1 [cbcb01fe]: Starting read_QC_trim_pe.fastq_scan_clean [2024-01-10 14:07:24,55] [info] Assigned new job execution tokens to the following groups: f226a562: 1 [2024-01-10 14:07:24,56] [warn] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.fastq_scan_clean:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:07:24,57] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_clean/inputs/-483590200/SRR13687078_1.clean.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/execution/SRR13687078_1.clean.fastq.gz: Operation not permitted [2024-01-10 14:07:24,58] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_clean/inputs/-483590200/SRR13687078_2.clean.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/execution/SRR13687078_2.clean.fastq.gz: Operation not permitted [2024-01-10 14:07:24,58] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.fastq_scan_clean:NA:1]: # capture date and version date | tee DATE fastq-scan -v | tee VERSION # set cat command based on compression if [[ "/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_clean/inputs/-483590200/SRR13687078_1.clean.fastq.gz" == *".gz" ]] ; then cat_reads="zcat" else cat_reads="cat" fi # capture forward read stats eval "${cat_reads} /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_clean/inputs/-483590200/SRR13687078_1.clean.fastq.gz" | fastq-scan | tee SRR13687078_1.clean_fastq-scan.json cat SRR13687078_1.clean_fastq-scan.json | jq .qc_stats.read_total | tee READ1_SEQS read1_seqs=$(cat READ1_SEQS) eval "${cat_reads} /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_clean/inputs/-483590200/SRR13687078_2.clean.fastq.gz" | fastq-scan | tee SRR13687078_2.clean_fastq-scan.json cat SRR13687078_2.clean_fastq-scan.json | jq .qc_stats.read_total | tee READ2_SEQS read2_seqs=$(cat READ2_SEQS) # capture number of read pairs if [ "${read1_seqs}" == "${read2_seqs}" ]; then read_pairs=${read1_seqs} else read_pairs="Uneven pairs: R1=${read1_seqs}, R2=${read2_seqs}" fi echo $read_pairs | tee READ_PAIRS [2024-01-10 14:07:24,61] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.fastq_scan_clean:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_clean/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_clean/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_clean:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_clean:delegated \ us-docker.pkg.dev/general-theiagen/biocontainers/fastq-scan@sha256:dd4880ca592ef724a15cd63dfcb4e3e04f5d4b269976e563ab7413d8ca5cfcfa /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_clean/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_clean/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-fastq_scan_clean/execution/docker_cid # return exit code exit $rc [2024-01-10 14:07:29,38] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.fastq_scan_clean:NA:1]: job id: 51157 [2024-01-10 14:07:29,38] [info] BackgroundConfigAsyncJobExecutionActor [cbcb01feread_QC_trim_pe.fastq_scan_clean:NA:1]: Status change from - to Done [2024-01-10 14:07:32,19] [info] cbcb01fe-5890-47ca-8ff3-c47190c0201c-SubWorkflowActor-SubWorkflow-read_QC_trim:-1:1 [cbcb01fe]: Workflow read_QC_trim_pe complete. Final Outputs: { "read_QC_trim_pe.kraken_version": "Kraken version 2.0.8-beta", "read_QC_trim_pe.kraken_target_org_dehosted": "", "read_QC_trim_pe.read1_human_spots_removed": 0, "read_QC_trim_pe.fastq_scan_clean1": 20648, "read_QC_trim_pe.fastq_scan_version": "fastq-scan 0.4.4", "read_QC_trim_pe.read1_clean": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/execution/SRR13687078_1.clean.fastq.gz", "read_QC_trim_pe.kraken_sc2_dehosted": 98.18, "read_QC_trim_pe.fastq_scan_clean_pairs": "20648", "read_QC_trim_pe.fastqc_docker": null, "read_QC_trim_pe.kraken_sc2": 98.18, "read_QC_trim_pe.kraken_report": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_raw/execution/SRR13687078_kraken2_report.txt", "read_QC_trim_pe.bbduk_docker": "us-docker.pkg.dev/general-theiagen/staphb/bbtools:38.76", "read_QC_trim_pe.kraken_human_dehosted": 0.03, "read_QC_trim_pe.fastqc_clean2": null, "read_QC_trim_pe.fastqc_raw2": null, "read_QC_trim_pe.fastqc_version": null, "read_QC_trim_pe.fastp_version": null, "read_QC_trim_pe.fastqc_clean2_html": null, "read_QC_trim_pe.read2_human_spots_removed": 0, "read_QC_trim_pe.midas_report": null, "read_QC_trim_pe.fastqc_clean1": null, "read_QC_trim_pe.fastqc_raw1": null, "read_QC_trim_pe.fastqc_clean1_html": null, "read_QC_trim_pe.read1_dehosted": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/execution/SRR13687078_R1_dehosted.fastq.gz", "read_QC_trim_pe.fastqc_raw1_html": null, "read_QC_trim_pe.midas_docker": null, "read_QC_trim_pe.read2_clean": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/execution/SRR13687078_2.clean.fastq.gz", "read_QC_trim_pe.midas_secondary_genus_abundance": null, "read_QC_trim_pe.kraken_report_dehosted": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-kraken2_theiacov_dehosted/execution/SRR13687078_kraken2_report.txt", "read_QC_trim_pe.fastq_scan_docker": "quay.io/biocontainers/fastq-scan:0.4.4--h7d875b9_1", "read_QC_trim_pe.fastq_scan_raw_pairs": "33826", "read_QC_trim_pe.fastq_scan_raw1": 33826, "read_QC_trim_pe.midas_secondary_genus": null, "read_QC_trim_pe.kraken_target_org": "", "read_QC_trim_pe.fastqc_raw_pairs": null, "read_QC_trim_pe.ncbi_scrub_docker": "us-docker.pkg.dev/general-theiagen/ncbi/sra-human-scrubber:1.0.2021-05-05", "read_QC_trim_pe.trimmomatic_version": "Trimmomatic 0.39", "read_QC_trim_pe.average_read_length": null, "read_QC_trim_pe.fastqc_clean_pairs": null, "read_QC_trim_pe.trimmomatic_docker": "us-docker.pkg.dev/general-theiagen/staphb/trimmomatic:0.39", "read_QC_trim_pe.fastq_scan_raw2": 33826, "read_QC_trim_pe.kraken_docker": "us-docker.pkg.dev/general-theiagen/staphb/kraken2:2.0.8-beta_hv", "read_QC_trim_pe.fastqc_raw2_html": null, "read_QC_trim_pe.read2_dehosted": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-ncbi_scrub_pe/execution/SRR13687078_R2_dehosted.fastq.gz", "read_QC_trim_pe.kraken_target_org_name": null, "read_QC_trim_pe.fastq_scan_clean2": 20648, "read_QC_trim_pe.kraken_human": 0.03, "read_QC_trim_pe.midas_primary_genus": null } [2024-01-10 14:07:36,20] [info] WorkflowExecutionActor-f226a562-09f6-40c5-8a8e-81c591f0be48 [f226a562]: Starting theiacov_illumina_pe.clean_check_reads [2024-01-10 14:07:44,55] [info] Assigned new job execution tokens to the following groups: f226a562: 1 [2024-01-10 14:07:44,56] [warn] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.clean_check_reads:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:07:44,56] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/inputs/-483590200/SRR13687078_1.clean.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/execution/SRR13687078_1.clean.fastq.gz: Operation not permitted [2024-01-10 14:07:44,57] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/inputs/-483590200/SRR13687078_2.clean.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/execution/SRR13687078_2.clean.fastq.gz: Operation not permitted [2024-01-10 14:07:44,58] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.clean_check_reads:NA:1]: flag="PASS" # initalize estimated genome size estimated_genome_size=0 if [[ "false" == "false" ]] ; then # set cat command based on compression if [[ "/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/inputs/-483590200/SRR13687078_1.clean.fastq.gz" == *".gz" ]] ; then cat_reads="zcat" else cat_reads="cat" fi # check one: number of reads read1_num=eval "$cat_reads /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/inputs/-483590200/SRR13687078_1.clean.fastq.gz" | awk '{s++}END{print s/4}' read2_num=eval "$cat_reads /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/inputs/-483590200/SRR13687078_2.clean.fastq.gz" | awk '{s++}END{print s/4}' # awk '{s++}END{print s/4' counts the number of lines and divides them by 4 # key assumption: in fastq there will be four lines per read # sometimes fastqs do not have 4 lines per read, so this might fail one day reads_total=$(expr $read1_num + $read2_num) if [ "${reads_total}" -le "113" ]; then flag="FAIL; the total number of reads is below the minimum of 113" else flag="PASS" fi # checks two and three: number of basepairs and proportion of sequence if [ "${flag}" == "PASS" ]; then # count number of basepairs # this only works if the fastq has 4 lines per read, so this might fail one day read1_bp=eval "${cat_reads} /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/inputs/-483590200/SRR13687078_1.clean.fastq.gz" | paste - - - - | cut -f2 | tr -d '\n' | wc -c read2_bp=eval "${cat_reads} /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/inputs/-483590200/SRR13687078_2.clean.fastq.gz" | paste - - - - | cut -f2 | tr -d '\n' | wc -c # paste - - - - (print 4 consecutive lines in one row, tab delimited) # cut -f2 print only the second column (the second line of the fastq 4-line) # tr -d '\n' removes line endings # wc -c counts characters # set proportion variables for easy comparison # removing the , 2) to make these integers instead of floats percent_read1=$(python3 -c "print(round(($read1_bp / ($read1_bp + $read2_bp))*100))") percent_read2=$(python3 -c "print(round(($read2_bp / ($read1_bp + $read2_bp))*100))") if [ "$percent_read1" -lt "40" ] ; then flag="FAIL; more than 40 percent of the total sequence is found in R2 (BP: $read2_bp; PERCENT: $percent_read2) compared to R1 (BP: $read1_bp; PERCENT: $percent_read1)" elif [ "$percent_read2" -lt "40" ] ; then flag="FAIL; more than 40 percent of the total sequence is found in R1 (BP: $read1_bp; PERCENT: $percent_read1) compared to R2 (BP: $read2_bp; PERCENT: $percent_read2)" else flag="PASS" fi # check total number of basepairs if [ "$flag" == "PASS" ] ; then bp_total=$(expr $read1_bp + $read2_bp) if [ "${bp_total}" -le "34000" ]; then flag="FAIL; the total number of basepairs is below the minimum of 34000" else flag="PASS" fi fi fi #checks four and five: estimated genome size and coverage if [ "${flag}" == "PASS" ]; then # estimate genome size if theiaprok AND expected_genome_size was not provided if [ "theiacov" == "theiaprok" ] && [[ -z "" ]]; then # First Pass; assuming average depth mash sketch -o test -k 31 -m 3 -r /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/inputs/-483590200/SRR13687078_1.clean.fastq.gz /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/inputs/-483590200/SRR13687078_2.clean.fastq.gz > mash-output.txt 2>&1 grep "Estimated genome size:" mash-output.txt | \ awk '{if($4){printf("%5.0f\n", $4)}} END {if (!NR) print "0"}' > genome_size_output grep "Estimated coverage:" mash-output.txt | \ awk '{if($3){printf("%d", $3)}} END {if (!NR) print "0"}' > coverage_output rm -rf test.msh rm -rf mash-output.txt estimated_genome_size=head -n1 genome_size_output estimated_coverage=head -n1 coverage_output # Check if second pass is needed if [ ${estimated_genome_size} -gt "2673870" ] || [ ${estimated_genome_size} -lt "1700" ] ; then # Probably high coverage, try increasing number of kmer copies to 10 M="-m 10" if [ ${estimated_genome_size} -lt "1700" ]; then # Probably low coverage, try decreasing the number of kmer copies to 1 M="-m 1" fi mash sketch -o test -k 31 ${M} -r /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/inputs/-483590200/SRR13687078_1.clean.fastq.gz /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/inputs/-483590200/SRR13687078_2.clean.fastq.gz > mash-output.txt 2>&1 grep "Estimated genome size:" mash-output.txt | \ awk '{if($4){printf("%5.0f\n", $4)}} END {if (!NR) print "0"}' > genome_size_output grep "Estimated coverage:" mash-output.txt | \ awk '{if($3){printf("%d", $3)}} END {if (!NR) print "0"}' > coverage_output rm -rf test.msh rm -rf mash-output.txt fi estimated_genome_size=head -n1 genome_size_output estimated_coverage=head -n1 coverage_output # estimate coverage if theiacov OR expected_genome_size was provided elif [ "theiacov" == "theiacov" ] || [ "" ]; then if [ "" ]; then estimated_genome_size= # use user-provided expected_genome_size elif [ "sars-cov-2" == "sars-cov-2" ]; then estimated_genome_size=29903 # size taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_045512.2 elif [ "sars-cov-2" == "MPXV" ]; then estimated_genome_size=197209 # size of 2022 virus taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_063383.1 elif [ "sars-cov-2" == "flu" ]; then estimated_genome_size=14000 # 500 bp over the CDC's approximate full genome size of 13500 (see https://www.cdc.gov/flu/about/professionals/genetic-characterization.htm) elif [ "sars-cov-2" == "HIV" ]; then estimated_genome_size=9181 # size taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_001802.1 elif [ "sars-cov-2" == "WNV" ]; then estimated_genome_size=11092 # WNV lineage 1 size from https://www.ncbi.nlm.nih.gov/nuccore/NC_009942.1 else flag="FAIL; the organism tag provided (sars-cov-2) is not valid and no expected_genome_size was provided." fi # coverage is calculated here by N/G where N is number of bases, and G is genome length # this will nearly always be an overestimation if [ $estimated_genome_size -ne 0 ]; then # prevent divided by zero errors estimated_coverage=$(python3 -c "print(round(($read1_bp+$read2_bp)/$estimated_genome_size))") else # they provided 0 for estimated_genome_size, nice estimated_coverage=0 fi else # workflow series was not provided or no est genome size was provided; default to fail estimated_genome_size=0 estimated_coverage=0 fi if [ "${estimated_genome_size}" -ge "2673870" ] && [ "theiacov" == "theiaprok" ] ; then flag="FAIL; the estimated genome size (${estimated_genome_size}) is larger than the maximum of 2673870 bps" elif [ "${estimated_genome_size}" -le "1700" ] && [ "theiacov" == "theiaprok" ] ; then flag="FAIL; the estimated genome size (${estimated_genome_size}) is smaller than the minimum of 1700 bps" else flag="PASS" if [ "${estimated_coverage}" -lt "10" ] ; then flag="FAIL; the estimated coverage (${estimated_coverage}) is less than the minimum of 10x" else flag="PASS" echo $estimated_genome_size | tee EST_GENOME_LENGTH fi fi fi fi echo $flag | tee FLAG echo $estimated_genome_size | tee EST_GENOME_LENGTH [2024-01-10 14:07:44,59] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.clean_check_reads:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads:delegated \ us-docker.pkg.dev/general-theiagen/bactopia/gather_samples@sha256:2e044c7be68519b9d683cd242177518e6db662b40b3bf669f891954f90a4d9a5 /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-clean_check_reads/execution/docker_cid # return exit code exit $rc [2024-01-10 14:07:49,38] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.clean_check_reads:NA:1]: job id: 51436 [2024-01-10 14:07:49,38] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.clean_check_reads:NA:1]: Status change from - to Done [2024-01-10 14:07:56,61] [info] c1c33d82-59e0-459f-8f06-9a7d398dfd47-SubWorkflowActor-SubWorkflow-ivar_consensus:-1:1 [c1c33d82]: Starting ivar_consensus.bwa [2024-01-10 14:08:04,55] [info] Assigned new job execution tokens to the following groups: f226a562: 1 [2024-01-10 14:08:04,79] [warn] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.bwa:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:08:04,79] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/inputs/-483590200/SRR13687078_2.clean.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/execution/SRR13687078_2.clean.fastq.gz: Operation not permitted [2024-01-10 14:08:04,80] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/inputs/-483590200/SRR13687078_1.clean.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-read_QC_trim/read_QC_trim_pe/cbcb01fe-5890-47ca-8ff3-c47190c0201c/call-bbduk/execution/SRR13687078_1.clean.fastq.gz: Operation not permitted [2024-01-10 14:08:04,80] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.bwa:NA:1]: # date and version control date | tee DATE echo "BWA $(bwa 2>&1 | grep Version )" | tee BWA_VERSION samtools --version | head -n1 | tee SAMTOOLS_VERSION # set reference genome if [[ ! -z "" ]]; then echo "User reference identified; will be utilized for alignement" ref_genome="" bwa index "" # move to primer_schemes dir; bwa fails if reference file not in this location else ref_genome="/artic-ncov2019/primer_schemes/nCoV-2019/V3/nCoV-2019.reference.fasta" fi # Map with BWA MEM echo "Running bwa mem -t 6 ${ref_genome} /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/inputs/-483590200/SRR13687078_1.clean.fastq.gz /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/inputs/-483590200/SRR13687078_2.clean.fastq.gz | samtools sort | samtools view -F 4 -o SRR13687078.sorted.bam " bwa mem \ -t 6 \ "${ref_genome}" \ /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/inputs/-483590200/SRR13687078_1.clean.fastq.gz /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/inputs/-483590200/SRR13687078_2.clean.fastq.gz |\ samtools sort | samtools view -F 4 -o SRR13687078.sorted.bam if [[ ! -z "/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/inputs/-483590200/SRR13687078_2.clean.fastq.gz" ]]; then echo "processing paired reads" samtools fastq -F4 -1 SRR13687078_R1.fastq.gz -2 SRR13687078_R2.fastq.gz SRR13687078.sorted.bam else echo "processing single-end reads" samtools fastq -F4 SRR13687078.sorted.bam | gzip > SRR13687078_R1.fastq.gz fi # index BAMs samtools index SRR13687078.sorted.bam [2024-01-10 14:08:04,82] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.bwa:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa:delegated \ us-docker.pkg.dev/general-theiagen/staphb/ivar@sha256:3c3cb44b0db91912aa5b895c2bd9b7af71feebbae8dc959f0466c21b3d61c40b /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/execution/docker_cid # return exit code exit $rc [2024-01-10 14:08:09,37] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.bwa:NA:1]: job id: 51706 [2024-01-10 14:08:09,37] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.bwa:NA:1]: Status change from - to Done [2024-01-10 14:08:10,88] [info] c1c33d82-59e0-459f-8f06-9a7d398dfd47-SubWorkflowActor-SubWorkflow-ivar_consensus:-1:1 [c1c33d82]: Starting ivar_consensus.primer_trim, ivar_consensus.stats_n_coverage [2024-01-10 14:08:14,55] [info] Assigned new job execution tokens to the following groups: f226a562: 2 [2024-01-10 14:08:14,56] [warn] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.primer_trim:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:08:14,57] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim/inputs/1413016897/SRR13687078.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/execution/SRR13687078.sorted.bam: Operation not permitted [2024-01-10 14:08:14,57] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.primer_trim:NA:1]: # date and version control echo "artic-v3.primers.bed" | tee PRIMER_NAME date | tee DATE ivar version | head -n1 | tee IVAR_VERSION samtools --version | head -n1 | tee SAMTOOLS_VERSION # trimming primers ivar trim \ -e \ -i /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim/inputs/1413016897/SRR13687078.sorted.bam \ -b /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim/inputs/1063836572/artic-v3.primers.bed \ -p SRR13687078.primertrim | tee IVAR_OUT # sorting and indexing the trimmed bams samtools sort \ SRR13687078.primertrim.bam \ -o SRR13687078.primertrim.sorted.bam samtools index SRR13687078.primertrim.sorted.bam PCT=$(grep "Trimmed primers from" IVAR_OUT | perl -lape 's/Trimmed primers from (\S+)%.*/$1/') echo $PCT if [[ $PCT = -* ]]; then echo 0; else echo $PCT; fi > IVAR_TRIM_PCT [2024-01-10 14:08:14,60] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.primer_trim:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim:delegated \ us-docker.pkg.dev/general-theiagen/staphb/ivar@sha256:3c3cb44b0db91912aa5b895c2bd9b7af71feebbae8dc959f0466c21b3d61c40b /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim/execution/docker_cid # return exit code exit $rc [2024-01-10 14:08:14,74] [warn] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.stats_n_coverage:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:08:14,75] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage/inputs/1413016897/SRR13687078.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/execution/SRR13687078.sorted.bam: Operation not permitted [2024-01-10 14:08:14,76] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.stats_n_coverage:NA:1]: date | tee DATE samtools --version | head -n1 | tee VERSION samtools stats /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage/inputs/1413016897/SRR13687078.sorted.bam > SRR13687078.stats.txt samtools coverage /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage/inputs/1413016897/SRR13687078.sorted.bam -m -o SRR13687078.cov.hist samtools coverage /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage/inputs/1413016897/SRR13687078.sorted.bam -o SRR13687078.cov.txt samtools flagstat /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage/inputs/1413016897/SRR13687078.sorted.bam > SRR13687078.flagstat.txt coverage=$(cut -f 6 SRR13687078.cov.txt | tail -n 1) depth=$(cut -f 7 SRR13687078.cov.txt | tail -n 1) meanbaseq=$(cut -f 8 SRR13687078.cov.txt | tail -n 1) meanmapq=$(cut -f 9 SRR13687078.cov.txt | tail -n 1) if [ -z "$coverage" ] ; then coverage="0" ; fi if [ -z "$depth" ] ; then depth="0" ; fi if [ -z "$meanbaseq" ] ; then meanbaseq="0" ; fi if [ -z "$meanmapq" ] ; then meanmapq="0" ; fi echo $coverage | tee COVERAGE echo $depth | tee DEPTH echo $meanbaseq | tee MEANBASEQ echo $meanmapq | tee MEANMAPQ [2024-01-10 14:08:14,80] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.stats_n_coverage:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage:delegated \ us-docker.pkg.dev/general-theiagen/staphb/samtools@sha256:ef92cb5e490c75e66e89ab0a113865839e6954435d363af400eaa27177ff7c91 /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage/execution/docker_cid # return exit code exit $rc [2024-01-10 14:08:19,38] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.primer_trim:NA:1]: job id: 51948 [2024-01-10 14:08:19,38] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.stats_n_coverage:NA:1]: job id: 52027 [2024-01-10 14:08:19,39] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.stats_n_coverage:NA:1]: Status change from - to Done [2024-01-10 14:08:19,39] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.primer_trim:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:08:22,35] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.primer_trim:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:08:24,14] [info] c1c33d82-59e0-459f-8f06-9a7d398dfd47-SubWorkflowActor-SubWorkflow-ivar_consensus:-1:1 [c1c33d82]: Starting ivar_consensus.stats_n_coverage_primtrim [2024-01-10 14:08:24,55] [info] Assigned new job execution tokens to the following groups: f226a562: 1 [2024-01-10 14:08:24,56] [warn] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.stats_n_coverage_primtrim:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:08:24,56] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage_primtrim/inputs/-1795192029/SRR13687078.primertrim.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim/execution/SRR13687078.primertrim.sorted.bam: Operation not permitted [2024-01-10 14:08:24,57] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.stats_n_coverage_primtrim:NA:1]: date | tee DATE samtools --version | head -n1 | tee VERSION samtools stats /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage_primtrim/inputs/-1795192029/SRR13687078.primertrim.sorted.bam > SRR13687078.stats.txt samtools coverage /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage_primtrim/inputs/-1795192029/SRR13687078.primertrim.sorted.bam -m -o SRR13687078.cov.hist samtools coverage /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage_primtrim/inputs/-1795192029/SRR13687078.primertrim.sorted.bam -o SRR13687078.cov.txt samtools flagstat /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage_primtrim/inputs/-1795192029/SRR13687078.primertrim.sorted.bam > SRR13687078.flagstat.txt coverage=$(cut -f 6 SRR13687078.cov.txt | tail -n 1) depth=$(cut -f 7 SRR13687078.cov.txt | tail -n 1) meanbaseq=$(cut -f 8 SRR13687078.cov.txt | tail -n 1) meanmapq=$(cut -f 9 SRR13687078.cov.txt | tail -n 1) if [ -z "$coverage" ] ; then coverage="0" ; fi if [ -z "$depth" ] ; then depth="0" ; fi if [ -z "$meanbaseq" ] ; then meanbaseq="0" ; fi if [ -z "$meanmapq" ] ; then meanmapq="0" ; fi echo $coverage | tee COVERAGE echo $depth | tee DEPTH echo $meanbaseq | tee MEANBASEQ echo $meanmapq | tee MEANMAPQ [2024-01-10 14:08:24,60] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.stats_n_coverage_primtrim:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage_primtrim/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage_primtrim/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage_primtrim:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage_primtrim:delegated \ us-docker.pkg.dev/general-theiagen/staphb/samtools@sha256:ef92cb5e490c75e66e89ab0a113865839e6954435d363af400eaa27177ff7c91 /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage_primtrim/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage_primtrim/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage_primtrim/execution/docker_cid # return exit code exit $rc [2024-01-10 14:08:25,16] [info] c1c33d82-59e0-459f-8f06-9a7d398dfd47-SubWorkflowActor-SubWorkflow-ivar_consensus:-1:1 [c1c33d82]: Starting ivar_consensus.consensus, ivar_consensus.variant_call [2024-01-10 14:08:29,37] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.stats_n_coverage_primtrim:NA:1]: job id: 52363 [2024-01-10 14:08:29,37] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.stats_n_coverage_primtrim:NA:1]: Status change from - to Done [2024-01-10 14:08:34,55] [info] Assigned new job execution tokens to the following groups: f226a562: 2 [2024-01-10 14:08:34,56] [warn] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.consensus:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:08:34,56] [warn] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.variant_call:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:08:34,56] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-consensus/inputs/-1795192029/SRR13687078.primertrim.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim/execution/SRR13687078.primertrim.sorted.bam: Operation not permitted [2024-01-10 14:08:34,57] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-variant_call/inputs/-1795192029/SRR13687078.primertrim.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim/execution/SRR13687078.primertrim.sorted.bam: Operation not permitted [2024-01-10 14:08:34,57] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.consensus:NA:1]: # date and version control date | tee DATE ivar version | head -n1 | tee IVAR_VERSION samtools --version | head -n1 | tee SAMTOOLS_VERSION # set reference genome if [[ ! -z "" ]]; then echo "User reference identified; will be utilized for alignement" ref_genome="" bwa index "" # move to primer_schemes dir; bwa fails if reference file not in this location else ref_genome="/artic-ncov2019/primer_schemes/nCoV-2019/V3/nCoV-2019.reference.fasta" fi # call consensus samtools mpileup \ --count-orphans \ -d 600000 \ --no-BAQ \ -Q 0 \ --reference ${ref_genome} \ /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-consensus/inputs/-1795192029/SRR13687078.primertrim.sorted.bam | \ ivar consensus \ -p SRR13687078.consensus \ -q 20 \ -t 0.6 \ -m 100 \ -n N \ # clean up fasta header echo ">SRR13687078" > SRR13687078.ivar.consensus.fasta grep -v ">" SRR13687078.consensus.fa >> SRR13687078.ivar.consensus.fasta [2024-01-10 14:08:34,58] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.variant_call:NA:1]: # date and version control date | tee DATE ivar version | head -n1 | tee IVAR_VERSION samtools --version | head -n1 | tee SAMTOOLS_VERSION # set reference genome if [[ ! -z "" ]]; then echo "User reference identified; will be utilized for alignement" ref_genome="" bwa index "" # move to primer_schemes dir; bwa fails if reference file not in this location else ref_genome="/artic-ncov2019/primer_schemes/nCoV-2019/V3/nCoV-2019.reference.fasta" fi # set reference gff if [[ ! -z "" ]]; then echo "User reference identified; will be utilized for alignement" ref_gff="" # move to primer_schemes dir; bwa fails if reference file not in this location else ref_gff="/reference/GCF_009858895.2_ASM985889v3_genomic.gff" fi # call variants samtools mpileup \ -A \ -d 600000 \ -B \ -Q 0 \ --reference ${ref_genome} \ /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-variant_call/inputs/-1795192029/SRR13687078.primertrim.sorted.bam | \ ivar variants \ -p SRR13687078.variants \ -q 20 \ -t 0.6 \ -m 100 \ -r ${ref_genome} \ -g ${ref_gff} # Convert TSV to VCF ivar_variants_to_vcf.py SRR13687078.variants.tsv SRR13687078.variants.vcf # Variant calculations # keep only unique nucleotide substitutions: # the same nucleotide substitution can be listed in multiple rows if it is within # more than one coding region, so remove columns with coding region information # then keep only unique rows cut -f 1-14 SRR13687078.variants.tsv | awk '!seen[$0]++ {print}' > unique_variants.tsv # filter variants that pass fisher's exact test grep "TRUE" unique_variants.tsv > passed_variants.tsv # calculate total number of variants variants_num=$(cat passed_variants.tsv | wc -l) if [ -z "$variants_num" ] ; then variants_num="0" ; fi echo $variants_num | tee VARIANT_NUM # calculate proportion of variants with allele frequencies between 0.6 and 0.9 # find number of variants at intermediate frequencies awk -F "\t" '{ if(($11 >= 0.6) && ($11 <= 0.9)) {print }}' passed_variants.tsv > intermediate_variants.tsv intermediates_num=$(cat intermediate_variants.tsv | wc -l) if [ -z "$intermediates_num" ] ; then intermediates_num="0" ; fi # if number of total variants is not zero, divide number of intermediate variants by total number of variants if [[ "$variants_num" -eq "0" ]]; then echo "Not computed: no variants detected" > PROPORTION_INTERMEDIATE else echo $intermediates_num $variants_num | awk '{ print $1/$2 }' > PROPORTION_INTERMEDIATE fi [2024-01-10 14:08:34,59] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.consensus:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-consensus/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-consensus/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-consensus:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-consensus:delegated \ us-docker.pkg.dev/general-theiagen/staphb/ivar@sha256:3c3cb44b0db91912aa5b895c2bd9b7af71feebbae8dc959f0466c21b3d61c40b /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-consensus/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-consensus/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-consensus/execution/docker_cid # return exit code exit $rc [2024-01-10 14:08:34,60] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.variant_call:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-variant_call/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-variant_call/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-variant_call:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-variant_call:delegated \ us-docker.pkg.dev/general-theiagen/staphb/ivar@sha256:3c3cb44b0db91912aa5b895c2bd9b7af71feebbae8dc959f0466c21b3d61c40b /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-variant_call/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-variant_call/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-variant_call/execution/docker_cid # return exit code exit $rc [2024-01-10 14:08:39,38] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.variant_call:NA:1]: job id: 52617 [2024-01-10 14:08:39,39] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.consensus:NA:1]: job id: 52610 [2024-01-10 14:08:39,39] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.consensus:NA:1]: Status change from - to Done [2024-01-10 14:08:39,39] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.variant_call:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:08:46,54] [info] BackgroundConfigAsyncJobExecutionActor [c1c33d82ivar_consensus.variant_call:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:08:48,57] [info] c1c33d82-59e0-459f-8f06-9a7d398dfd47-SubWorkflowActor-SubWorkflow-ivar_consensus:-1:1 [c1c33d82]: Workflow ivar_consensus complete. Final Outputs: { "ivar_consensus.ivar_version_consensus": "iVar version 1.3.1", "ivar_consensus.read2_aligned": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/execution/SRR13687078_R2.fastq.gz", "ivar_consensus.assembly_method_nonflu": "BWA Version: 0.7.17-r1188; iVar version 1.3.1", "ivar_consensus.primer_bed_name": "artic-v3.primers.bed", "ivar_consensus.consensus_flagstat": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage/execution/SRR13687078.flagstat.txt", "ivar_consensus.ivar_variant_version": "iVar version 1.3.1", "ivar_consensus.samtools_version_primtrim": "samtools 1.12", "ivar_consensus.samtools_version_stats": "samtools 1.15", "ivar_consensus.primer_trimmed_read_percent": 32.11, "ivar_consensus.bwa_version": "BWA Version: 0.7.17-r1188", "ivar_consensus.consensus_stats": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-stats_n_coverage/execution/SRR13687078.stats.txt", "ivar_consensus.meanbaseq_trim": "38.4", "ivar_consensus.aligned_bam": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim/execution/SRR13687078.primertrim.sorted.bam", "ivar_consensus.samtools_version": "samtools 1.12", "ivar_consensus.meanmapq_trim": "60.0", "ivar_consensus.aligned_bai": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim/execution/SRR13687078.primertrim.sorted.bam.bai", "ivar_consensus.assembly_mean_coverage": "219.356", "ivar_consensus.ivar_vcf": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-variant_call/execution/SRR13687078.variants.vcf", "ivar_consensus.ivar_version_primtrim": "iVar version 1.3.1", "ivar_consensus.ivar_tsv": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-variant_call/execution/SRR13687078.variants.tsv", "ivar_consensus.ivar_variant_proportion_intermediate": "0", "ivar_consensus.consensus_n_variant_min_depth": 100, "ivar_consensus.read1_aligned": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-bwa/execution/SRR13687078_R1.fastq.gz", "ivar_consensus.samtools_version_consensus": "samtools 1.12", "ivar_consensus.assembly_fasta": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-consensus/execution/SRR13687078.ivar.consensus.fasta" } [2024-01-10 14:08:51,64] [info] WorkflowExecutionActor-f226a562-09f6-40c5-8a8e-81c591f0be48 [f226a562]: Starting theiacov_illumina_pe.consensus_qc, theiacov_illumina_pe.sc2_gene_coverage, theiacov_illumina_pe.nextclade, theiacov_illumina_pe.pangolin4 [2024-01-10 14:08:54,55] [info] Assigned new job execution tokens to the following groups: f226a562: 4 [2024-01-10 14:08:54,56] [warn] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.sc2_gene_coverage:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:08:54,57] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-primer_trim/execution/SRR13687078.primertrim.sorted.bam: Operation not permitted [2024-01-10 14:08:54,58] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.sc2_gene_coverage:NA:1]: samtools index /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam chr=$(samtools idxstats /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam | cut -f 1 | head -1) samtools coverage -r "${chr}:21563-25384" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam >> SRR13687078.cov.txt s_gene_depth=$(cut -f 7 SRR13687078.cov.txt | tail -n 1) # samtools outputs 3 columns; column 3 is the depth of coverage per nucleotide position, piped to awk to count the positions # above min_depth, then wc -l counts them all orf1ab=$(samtools depth -J -r "${chr}:266-21555" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) sgene=$(samtools depth -J -r "${chr}:21563-25384" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) orf3a=$(samtools depth -J -r "${chr}:25393-26220" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) egene=$(samtools depth -J -r "${chr}:26245-26472" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) mgene=$(samtools depth -J -r "${chr}:26523-27191" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) orf6=$(samtools depth -J -r "${chr}:27202-27387" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) orf7a=$(samtools depth -J -r "${chr}:27394-27759" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) orf7b=$(samtools depth -J -r "${chr}:27756-27887" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) orf8=$(samtools depth -J -r "${chr}:27894-28259" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) ngene=$(samtools depth -J -r "${chr}:28274-29533" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) orf10=$(samtools depth -J -r "${chr}:29558-29674" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/inputs/-1795192029/SRR13687078.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) orf1ab_pc=$(python3 -c "print ( round( ($orf1ab / 21290 ) * 100, 2 ) )") sgene_pc=$(python3 -c "print ( round( ($sgene / 3822 ) * 100, 2 ) )") orf3a_pc=$(python3 -c "print ( round( ($orf3a / 828 ) * 100, 2 ) )") egene_pc=$(python3 -c "print ( round( ($egene / 228 ) * 100, 2 ) )") mgene_pc=$(python3 -c "print ( round( ($mgene / 669 ) * 100, 2 ) )") orf6_pc=$(python3 -c "print ( round( ($orf6 / 186 ) * 100, 2 ) )") orf7a_pc=$(python3 -c "print ( round( ($orf7a / 366 ) * 100, 2 ) )") orf7b_pc=$(python3 -c "print ( round( ($orf7b / 132 ) * 100, 2 ) )") orf8_pc=$(python3 -c "print ( round( ($orf8 / 366 ) * 100, 2 ) )") ngene_pc=$(python3 -c "print ( round( ($ngene / 1260 ) * 100, 2 ) )") orf10_pc=$(python3 -c "print ( round( ($orf10 / 117 ) * 100, 2 ) )") echo -e "#NOTE: THE VALUES BELOW ASSUME WUHAN-1 REFERENCE GENOME" > SRR13687078.percent_gene_coverage.tsv echo -e "Gene\tPercent_Coverage" >> SRR13687078.percent_gene_coverage.tsv echo -e "ORF1ab\t" $orf1ab_pc >> SRR13687078.percent_gene_coverage.tsv echo -e "S_gene\t" $sgene_pc >> SRR13687078.percent_gene_coverage.tsv echo -e "ORF3a\t" $orf3a_pc >> SRR13687078.percent_gene_coverage.tsv echo -e "E_gene\t" $egene_pc >> SRR13687078.percent_gene_coverage.tsv echo -e "M_gene\t" $mgene_pc >> SRR13687078.percent_gene_coverage.tsv echo -e "ORF6\t" $orf6_pc >> SRR13687078.percent_gene_coverage.tsv echo -e "ORF7a\t" $orf7a_pc >> SRR13687078.percent_gene_coverage.tsv echo -e "ORF7b\t" $orf7b_pc >> SRR13687078.percent_gene_coverage.tsv echo -e "ORF8\t" $orf8_pc >> SRR13687078.percent_gene_coverage.tsv echo -e "N_gene\t" $ngene_pc >> SRR13687078.percent_gene_coverage.tsv echo -e "ORF10\t" $orf10_pc >> SRR13687078.percent_gene_coverage.tsv if [ -z "s_gene_depth" ] ; then s_gene_depth="0"; fi echo $s_gene_depth | tee S_GENE_DEPTH echo $sgene_pc | tee S_GENE_PC [2024-01-10 14:08:54,59] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.sc2_gene_coverage:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage:delegated \ us-docker.pkg.dev/general-theiagen/staphb/samtools@sha256:ef92cb5e490c75e66e89ab0a113865839e6954435d363af400eaa27177ff7c91 /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-sc2_gene_coverage/execution/docker_cid # return exit code exit $rc [2024-01-10 14:08:54,74] [warn] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.consensus_qc:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:08:54,75] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-consensus_qc/inputs/1958973516/SRR13687078.ivar.consensus.fasta -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-consensus/execution/SRR13687078.ivar.consensus.fasta: Operation not permitted [2024-01-10 14:08:54,76] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.consensus_qc:NA:1]: if [ -s "" ] ; then GENOME_LEN=$(grep -v ">" | tr --delete '\n' | wc -c) elif [ ] ; then GENOME_LEN= else # set SC2 default GENOME_LEN=29903 fi # capture date and version date | tee DATE num_N=$( grep -v ">" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-consensus_qc/inputs/1958973516/SRR13687078.ivar.consensus.fasta | grep -o 'N' | wc -l ) if [ -z "$num_N" ] ; then num_N="0" ; fi echo $num_N | tee NUM_N num_ACTG=$( grep -v ">" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-consensus_qc/inputs/1958973516/SRR13687078.ivar.consensus.fasta | grep -o -E "C|A|T|G" | wc -l ) if [ -z "$num_ACTG" ] ; then num_ACTG="0" ; fi echo $num_ACTG | tee NUM_ACTG # calculate percent coverage (Wu Han-1 genome length: 29903bp) python3 -c "print ( round( ($num_ACTG / $GENOME_LEN ) * 100, 2 ) )" | tee PERCENT_REF_COVERAGE num_degenerate=$( grep -v ">" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-consensus_qc/inputs/1958973516/SRR13687078.ivar.consensus.fasta | grep -o -E "B|D|E|F|H|I|J|K|L|M|O|P|Q|R|S|U|V|W|X|Y|Z" | wc -l ) if [ -z "$num_degenerate" ] ; then num_degenerate="0" ; fi echo $num_degenerate | tee NUM_DEGENERATE num_total=$( grep -v ">" /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-consensus_qc/inputs/1958973516/SRR13687078.ivar.consensus.fasta | grep -o -E '[A-Z]' | wc -l ) if [ -z "$num_total" ] ; then num_total="0" ; fi echo $num_total | tee NUM_TOTAL [2024-01-10 14:08:54,76] [warn] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.pangolin4:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:08:54,77] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-pangolin4/inputs/1958973516/SRR13687078.ivar.consensus.fasta -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-consensus/execution/SRR13687078.ivar.consensus.fasta: Operation not permitted [2024-01-10 14:08:54,77] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.pangolin4:NA:1]: set -e # date and version capture date | tee DATE { pangolin --all-versions && usher --version; } | tr '\n' ';' | cut -f -6 -d ';' | tee VERSION_PANGOLIN_ALL pangolin "/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-pangolin4/inputs/1958973516/SRR13687078.ivar.consensus.fasta" \ \ --min-length 10000 \ --max-ambig 0.5 \ --expanded-lineage \ \ \ --outfile "SRR13687078.pangolin_report.csv" \ --verbose \ python3 < /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-consensus/execution/SRR13687078.ivar.consensus.fasta: Operation not permitted [2024-01-10 14:08:54,98] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.nextclade:NA:1]: NEXTCLADE_VERSION="$(nextclade --version)" echo $NEXTCLADE_VERSION > NEXTCLADE_VERSION nextclade dataset get --name="sars-cov-2" --reference="MN908947" --tag="2023-09-21T12:00:00Z" -o nextclade_dataset_dir --verbose set -e nextclade run \ --input-dataset=nextclade_dataset_dir/ \ \ \ \ \ \ \ --output-json "SRR13687078.ivar.consensus".nextclade.json \ --output-tsv "SRR13687078.ivar.consensus".nextclade.tsv \ --output-tree "SRR13687078.ivar.consensus".nextclade.auspice.json \ --output-all=. \ "/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-nextclade/inputs/1958973516/SRR13687078.ivar.consensus.fasta" [2024-01-10 14:08:55,00] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.nextclade:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-nextclade/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-nextclade/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-nextclade:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-nextclade:delegated \ nextstrain/nextclade@sha256:f4440021c7f854433bd56433024f38199de27e0ef657e11d5fb28aad4f265a48 /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-nextclade/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-nextclade/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-nextclade/execution/docker_cid # return exit code exit $rc [2024-01-10 14:08:59,39] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.sc2_gene_coverage:NA:1]: job id: 53041 [2024-01-10 14:08:59,39] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.consensus_qc:NA:1]: job id: 53121 [2024-01-10 14:08:59,39] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.nextclade:NA:1]: job id: 53174 [2024-01-10 14:08:59,39] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.sc2_gene_coverage:NA:1]: Status change from - to Done [2024-01-10 14:08:59,39] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.pangolin4:NA:1]: job id: 53140 [2024-01-10 14:08:59,39] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.consensus_qc:NA:1]: Status change from - to Done [2024-01-10 14:08:59,39] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.nextclade:NA:1]: Status change from - to Done [2024-01-10 14:08:59,39] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.pangolin4:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:09:00,81] [info] WorkflowExecutionActor-f226a562-09f6-40c5-8a8e-81c591f0be48 [f226a562]: Starting theiacov_illumina_pe.vadr, theiacov_illumina_pe.nextclade_output_parser [2024-01-10 14:09:04,54] [info] Assigned new job execution tokens to the following groups: f226a562: 2 [2024-01-10 14:09:04,70] [warn] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.vadr:NA:1]: Unrecognized runtime attribute keys: dx_instance_type, cpu, memory [2024-01-10 14:09:04,72] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-vadr/inputs/1958973516/SRR13687078.ivar.consensus.fasta -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-ivar_consensus/ivar_consensus/c1c33d82-59e0-459f-8f06-9a7d398dfd47/call-consensus/execution/SRR13687078.ivar.consensus.fasta: Operation not permitted [2024-01-10 14:09:04,72] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.vadr:NA:1]: set -e if [ 24908 -gt 10000 ]; then # remove terminal ambiguous nucleotides /opt/vadr/vadr/miniscripts/fasta-trim-terminal-ambigs.pl \ "/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-vadr/inputs/1958973516/SRR13687078.ivar.consensus.fasta" \ --minlen 50 \ --maxlen 30000 \ > "SRR13687078.ivar.consensus_trimmed.fasta" # run VADR # --split and --cpu must be used in conjuction v-annotate.pl \ --split --cpu 2 \ --noseqnamemax --glsearch -s -r --nomisc --mkey sarscov2 --lowsim5seq 6 --lowsim3seq 6 --alt_fail lowscore,insertnn,deletinn --out_allfasta \ "SRR13687078.ivar.consensus_trimmed.fasta" \ "SRR13687078.ivar.consensus" # package everything for output tar -C "SRR13687078.ivar.consensus" -czvf "SRR13687078.ivar.consensus.vadr.tar.gz" . # package up FASTA files into zip file for output. Note: this will work whether the --out_allfasta flag is included or not (there are just more when the option is used) mkdir -v vadr_fasta_files cp -v SRR13687078.ivar.consensus/*.fa vadr_fasta_files zip SRR13687078.ivar.consensus_vadr-fasta-files.zip vadr_fasta_files/*.fa # prep alerts into a tsv file for parsing cut -f 5 "SRR13687078.ivar.consensus/SRR13687078.ivar.consensus.vadr.alt.list" | tail -n +2 > "SRR13687078.ivar.consensus.vadr.alerts.tsv" cat "SRR13687078.ivar.consensus.vadr.alerts.tsv" | wc -l > NUM_ALERTS else echo "VADR skipped due to poor assembly; assembly length (unambiguous) = 24908" > NUM_ALERTS fi [2024-01-10 14:09:04,73] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.vadr:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-vadr/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-vadr/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-vadr:/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-vadr:delegated \ us-docker.pkg.dev/general-theiagen/staphb/vadr@sha256:f15a73a6f6e0802c3a9c05e7c602740e7029a2e1ccc14aa6935b459be87c98d7 /cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-vadr/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-vadr/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-vadr/execution/docker_cid # return exit code exit $rc [2024-01-10 14:09:04,84] [warn] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.nextclade_output_parser:NA:1]: Unrecognized runtime attribute keys: disk, dx_instance_type, disks, cpu, memory [2024-01-10 14:09:04,84] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-nextclade_output_parser/inputs/210698167/SRR13687078.ivar.consensus.nextclade.tsv -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-nextclade/execution/SRR13687078.ivar.consensus.nextclade.tsv: Operation not permitted [2024-01-10 14:09:04,85] [info] BackgroundConfigAsyncJobExecutionActor [f226a562theiacov_illumina_pe.nextclade_output_parser:NA:1]: # Set WDL input variable to input.tsv file cat "/cromwell-executions/theiacov_illumina_pe/f226a562-09f6-40c5-8a8e-81c591f0be48/call-nextclade_output_parser/inputs/210698167/SRR13687078.ivar.consensus.nextclade.tsv" > input.tsv touch TAMIFLU_AASUBS # Parse outputs using python3 python3 <

✅ for theiacov_illumina_pe run via cromwell

kapsakcj commented 9 months ago

TheiaCov_Illumina_SE run via cromwell:

cromwell command used:

cromwell run -i ./tests/inputs/theiacov/wf_theiacov_illumina_se.json -m metadata.json ./workflows/theiacov/wf_theiacov_illumina_se.wdl
successful workflow output (again only caught some of the output due to scrollback buffer): ```Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/metadata_typed_values.xml::metadata_typed_values::tjeandet ran successfully in 4ms Running Changeset: changesets/workflow_metadata_summary.xml::metadata_journal_id_int_to_big_int::kshakir Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: METADATA_JOURNAL.METADATA_JOURNAL_ID datatype was changed to BIGINT Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Auto-increment added to METADATA_JOURNAL.METADATA_JOURNAL_ID Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::metadata_journal_id_int_to_big_int::kshakir ran successfully in 5ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_summary::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY created Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_summary::mcovarr ran successfully in 4ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_uuid_idx::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_METADATA_SUMMARY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_uuid_idx::mcovarr ran successfully in 3ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_name_idx::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_NAME_IDX created Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_name_idx::mcovarr ran successfully in 2ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_status_idx::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_STATUS_IDX created Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_status_idx::mcovarr ran successfully in 3ms Running Changeset: changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_subsecond_timestamp::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: METADATA_JOURNAL.METADATA_TIMESTAMP datatype was changed to DATETIME(6) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_subsecond_timestamp::mcovarr ran successfully in 3ms Running Changeset: changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_timestamp_not_null::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: NOT NULL constraint has been added to METADATA_JOURNAL.METADATA_TIMESTAMP Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_timestamp_not_null::mcovarr ran successfully in 2ms Running Changeset: changesets/workflow_store.xml::WORKFLOW_STORE::chrisl Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Table WORKFLOW_STORE created Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store.xml::WORKFLOW_STORE::chrisl ran successfully in 4ms Running Changeset: changesets/workflow_store.xml::workflow_store_uuid_index::kshakir Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_STORE(WORKFLOW_UUID) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store.xml::workflow_store_uuid_index::kshakir ran successfully in 5ms Running Changeset: changesets/workflow_store.xml::workflow_store_state_index::chrisl Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Index WORKFLOW_STORE_STATE_IDX created Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store.xml::workflow_store_state_index::chrisl ran successfully in 3ms Running Changeset: changesets/backend_KV_Store.xml::BACKEND_KV_STORE::rmunshi Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Table BACKEND_KV_STORE created Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/backend_KV_Store.xml::BACKEND_KV_STORE::rmunshi ran successfully in 3ms Running Changeset: changesets/backend_KV_Store.xml::backend_KV_store_job_key_constraint::rmunshi Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Unique constraint added to BACKEND_KV_STORE(WORKFLOW_EXECUTION_UUID, CALL_FQN, JOB_SCATTER_INDEX, JOB_RETRY_ATTEMPT, STORE_KEY) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/backend_KV_Store.xml::backend_KV_store_job_key_constraint::rmunshi ran successfully in 4ms Running Changeset: changesets/job_store.xml::JOB_STORE::chrisl Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Table JOB_STORE created Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/job_store.xml::JOB_STORE::chrisl ran successfully in 4ms Running Changeset: changesets/job_store.xml::job_store_uuid_index::chrisl Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Index JOB_STORE_UUID_IDX created Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/job_store.xml::job_store_uuid_index::chrisl ran successfully in 2ms Running Changeset: changesets/job_store.xml::job_store_jobkey_index::chrisl Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE(WORKFLOW_UUID, CALL_FQN, JOB_SCATTER_INDEX, JOB_RETRY_ATTEMPT) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/job_store.xml::job_store_jobkey_index::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_metainfo::chrisl Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_METAINFO created Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_metainfo::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_metainfo_uniqueness::chrisl Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_RESULT_METAINFO(WORKFLOW_UUID, CALL_FQN, JOB_SCATTER_INDEX) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_metainfo_uniqueness::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_hashes::chrisl Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Table CALL_CACHING_HASH created Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_hashes::chrisl ran successfully in 2ms Running Changeset: changesets/callcaching.xml::call_caching_result_hashes_uniqueness::chrisl Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_HASH(HASH_KEY, RESULT_METAINFO_ID) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_hashes_uniqueness::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_hashes_uniqueness_foreign_key::chrisl Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH (RESULT_METAINFO_ID) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_hashes_uniqueness_foreign_key::chrisl ran successfully in 2ms Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons::chrisl Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_SIMPLETON created Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_simpletons::chrisl ran successfully in 2ms Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons_uniqueness::chrisl Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_RESULT_SIMPLETON(SIMPLETON_KEY, RESULT_METAINFO_ID) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_simpletons_uniqueness::chrisl ran successfully in 2ms Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons_foreign_key::chrisl Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_RESULT_SIMPLETON (RESULT_METAINFO_ID) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_simpletons_foreign_key::chrisl ran successfully in 2ms Running Changeset: changesets/call_caching_allow_result_reuse_fix.xml::call_caching_fix_boolean::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: CALL_CACHING_RESULT_METAINFO.ALLOW_RESULT_REUSE datatype was changed to BOOLEAN Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_allow_result_reuse_fix.xml::call_caching_fix_boolean::mcovarr ran successfully in 1ms Running Changeset: changesets/call_caching_allow_result_reuse_fix.xml::call_caching_reuse_default::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Default value added to CALL_CACHING_RESULT_METAINFO.ALLOW_RESULT_REUSE Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_allow_result_reuse_fix.xml::call_caching_reuse_default::mcovarr ran successfully in 1ms Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Table JOB_STORE_RESULT_SIMPLETON created Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_result_simpletons::mcovarr ran successfully in 3ms Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons_uniqueness::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE_RESULT_SIMPLETON(SIMPLETON_KEY, JOB_STORE_ID) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_result_simpletons_uniqueness::mcovarr ran successfully in 2ms Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons_foreign_key::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_RESULT_SIMPLETON (JOB_STORE_ID) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_result_simpletons_foreign_key::mcovarr ran successfully in 2ms Running Changeset: changesets/job_store_simpletons.xml::job_store_remove_job_output::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Column JOB_STORE.JOB_OUTPUT dropped Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_remove_job_output::mcovarr ran successfully in 2ms Running Changeset: changesets/restart_and_recover_migration.xml::WORKFLOW_STORE_MIGRATION::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::WORKFLOW_STORE_MIGRATION::mcovarr ran successfully in 4ms Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_MIGRATION::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::JOB_STORE_MIGRATION::mcovarr ran successfully in 7ms Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_VALUE_NULLABLE::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Null constraint dropped from JOB_STORE_RESULT_SIMPLETON.SIMPLETON_VALUE Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_VALUE_NULLABLE::mcovarr ran successfully in 21ms Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_MIGRATION::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: WORKFLOW_EXECUTION + EXECUTION + SYMBOL + JOB_STORE -> JOB_STORE_RESULT_SIMPLETON migration complete. Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_MIGRATION::mcovarr ran successfully in 3ms Running Changeset: changesets/restart_and_recover_migration.xml::BACKEND_KV_STORE_MIGRATION::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::BACKEND_KV_STORE_MIGRATION::mcovarr ran successfully in 4ms Running Changeset: changesets/restart_and_recover_migration.xml::WORKFLOW_OPTIONS_RENAME_MIGRATION::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Workflow option renaming migration complete. Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::WORKFLOW_OPTIONS_RENAME_MIGRATION::mcovarr ran successfully in 2ms Running Changeset: changesets/summary_status_table.xml::summary_status_table::kshakir Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Table SUMMARY_STATUS created Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/summary_status_table.xml::summary_status_table::kshakir ran successfully in 3ms Running Changeset: changesets/summary_status_table.xml::summary_status_summarized_table_name_index::kshakir Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS(SUMMARY_TABLE_NAME, SUMMARIZED_TABLE_NAME) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/summary_status_table.xml::summary_status_summarized_table_name_index::kshakir ran successfully in 2ms Running Changeset: changesets/standardize_column_names.xml::workflow_execution_uuid::rmunshi Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Column CALL_CACHING_RESULT_METAINFO.WORKFLOW_UUID renamed to WORKFLOW_EXECUTION_UUID Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Column WORKFLOW_STORE.WORKFLOW_UUID renamed to WORKFLOW_EXECUTION_UUID Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Column JOB_STORE.WORKFLOW_UUID renamed to WORKFLOW_EXECUTION_UUID Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names.xml::workflow_execution_uuid::rmunshi ran successfully in 4ms Running Changeset: changesets/standardize_column_names.xml::call_job_identifiers::rmunshi Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Column METADATA_JOURNAL.METADATA_CALL_FQN renamed to CALL_FQN Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Column METADATA_JOURNAL.METADATA_CALL_INDEX renamed to JOB_SCATTER_INDEX Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Column METADATA_JOURNAL.METADATA_CALL_ATTEMPT renamed to JOB_RETRY_ATTEMPT Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names.xml::call_job_identifiers::rmunshi ran successfully in 4ms Running Changeset: changesets/embiggen_metadata_value.xml::entry_or_journal_existence_xor::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/embiggen_metadata_value.xml::entry_or_journal_existence_xor::mcovarr ran successfully in 22ms Running Changeset: changesets/embiggen_metadata_value.xml::embiggen_metadata_entry::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Marking ChangeSet: changesets/embiggen_metadata_value.xml::embiggen_metadata_entry::mcovarr ran despite precondition failure due to onFail='MARK_RAN': changelog.xml : Table PUBLIC.METADATA_ENTRY does not exist Running Changeset: changesets/embiggen_metadata_value.xml::embiggen_metadata_journal::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: METADATA_JOURNAL.METADATA_VALUE datatype was changed to LONGTEXT Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/embiggen_metadata_value.xml::embiggen_metadata_journal::mcovarr ran successfully in 6ms Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus::rmunshi Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Table CALL_CACHING_JOB_DETRITUS created Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_job_detritus.xml::call_caching_job_detritus::rmunshi ran successfully in 3ms Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus_key_uniqueness::rmunshi Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_JOB_DETRITUS(JOB_DETRITUS_KEY, RESULT_METAINFO_ID) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_job_detritus.xml::call_caching_job_detritus_key_uniqueness::rmunshi ran successfully in 3ms Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus_foreign_key::rmunshi Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_JOB_DETRITUS (RESULT_METAINFO_ID) Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_job_detritus.xml::call_caching_job_detritus_foreign_key::rmunshi ran successfully in 3ms Running Changeset: changesets/job_store_tinyints.xml::job_store_fix_job_successful::mcovarr Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: JOB_STORE.JOB_SUCCESSFUL datatype was changed to BOOLEAN Jan 10, 2024 2:22:48 PM liquibase.changelog INFO: ChangeSet changesets/job_store_tinyints.xml::job_store_fix_job_successful::mcovarr ran successfully in 2ms Running Changeset: changesets/job_store_tinyints.xml::job_store_fix_job_retryable_failure::mcovarr Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: JOB_STORE.RETRYABLE_FAILURE datatype was changed to BOOLEAN Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/job_store_tinyints.xml::job_store_fix_job_retryable_failure::mcovarr ran successfully in 2ms Running Changeset: changesets/job_store_tinyints.xml::job_store_not_nullable_job_successful::mcovarr Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: NOT NULL constraint has been added to JOB_STORE.JOB_SUCCESSFUL Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/job_store_tinyints.xml::job_store_not_nullable_job_successful::mcovarr ran successfully in 2ms Running Changeset: changesets/standardize_column_names_again.xml::standardize_column_names_again::kshakir Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index JOB_STORE_UUID_IDX dropped from table JOB_STORE Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_NAME_IDX dropped from table WORKFLOW_METADATA_SUMMARY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_STATUS_IDX dropped from table WORKFLOW_METADATA_SUMMARY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index WORKFLOW_STORE_STATE_IDX dropped from table WORKFLOW_STORE Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_JOB_DETRITUS_KEY dropped from CALL_CACHING_JOB_DETRITUS Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_RESULT_METAINFO dropped from CALL_CACHING_RESULT_METAINFO Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_HASH dropped from CALL_CACHING_HASH Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_RESULT_SIMPLETON dropped from CALL_CACHING_RESULT_SIMPLETON Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint JOB_STORE_JOBKEY_IDX dropped from JOB_STORE Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint UK_BACKEND_KV_STORE_KEY dropped from BACKEND_KV_STORE Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint UK_JOB_STORE_RESULT_SIMPLETON dropped from JOB_STORE_RESULT_SIMPLETON Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint SUMMARY_STATUS_SUMMARY_TABLE_NAME_SUMMARIZED_TABLE_NAME_INDEX dropped from SUMMARY_STATUS Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint WORKFLOW_METADATA_UUID_IDX dropped from WORKFLOW_METADATA_SUMMARY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint WORKFLOW_STORE_UUID_IDX dropped from WORKFLOW_STORE Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key CCJD_RESULT_METAINFO_ID_FK dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key CCH_RESULT_METAINFO_ID_FK dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key CCRS_RESULT_METAINFO_ID_FK dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key JSRS_JOB_STORE_FK dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table CALL_CACHING_JOB_DETRITUS renamed to CALL_CACHING_DETRITUS_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_METAINFO renamed to CALL_CACHING_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table CALL_CACHING_HASH renamed to CALL_CACHING_HASH_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_SIMPLETON renamed to CALL_CACHING_SIMPLETON_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table BACKEND_KV_STORE renamed to JOB_KEY_VALUE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table JOB_STORE renamed to JOB_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table JOB_STORE_RESULT_SIMPLETON renamed to JOB_STORE_SIMPLETON_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table METADATA_JOURNAL renamed to METADATA_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table SUMMARY_STATUS renamed to SUMMARY_STATUS_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY renamed to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table WORKFLOW_STORE renamed to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_JOB_DETRITUS_ID renamed to CALL_CACHING_DETRITUS_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.JOB_DETRITUS_KEY renamed to DETRITUS_KEY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.JOB_DETRITUS_VALUE renamed to DETRITUS_VALUE Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column CALL_CACHING_ENTRY.CALL_CACHING_RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column CALL_CACHING_ENTRY.JOB_SCATTER_INDEX renamed to JOB_INDEX Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column CALL_CACHING_ENTRY.CALL_FQN renamed to CALL_FULLY_QUALIFIED_NAME Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column CALL_CACHING_HASH_ENTRY.RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ID renamed to CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column CALL_CACHING_SIMPLETON_ENTRY.RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_RESULT_SIMPLETON_ID renamed to CALL_CACHING_SIMPLETON_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.JOB_RETRY_ATTEMPT renamed to JOB_ATTEMPT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.JOB_SCATTER_INDEX renamed to JOB_INDEX Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.BACKEND_KV_STORE_ID renamed to JOB_KEY_VALUE_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.CALL_FQN renamed to CALL_FULLY_QUALIFIED_NAME Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.JOB_RETRY_ATTEMPT renamed to JOB_ATTEMPT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.JOB_SCATTER_INDEX renamed to JOB_INDEX Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.JOB_STORE_ID renamed to JOB_STORE_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.CALL_FQN renamed to CALL_FULLY_QUALIFIED_NAME Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_ID renamed to JOB_STORE_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_RESULT_SIMPLETON_ID renamed to JOB_STORE_SIMPLETON_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.SUMMARY_STATUS_ID renamed to SUMMARY_STATUS_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column WORKFLOW_METADATA_SUMMARY_ENTRY.END_DT renamed to END_TIMESTAMP Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column WORKFLOW_METADATA_SUMMARY_ENTRY.START_DT renamed to START_TIMESTAMP Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column WORKFLOW_METADATA_SUMMARY_ENTRY.WORKFLOW_METADATA_SUMMARY_ID renamed to WORKFLOW_METADATA_SUMMARY_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column WORKFLOW_STORE_ENTRY.WORKFLOW_STORE_ID renamed to WORKFLOW_STORE_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_DETRITUS_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_HASH_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_SIMPLETON_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key dropped from JOB_KEY_VALUE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key dropped from JOB_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key dropped from JOB_STORE_SIMPLETON_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key dropped from SUMMARY_STATUS_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key dropped from WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key dropped from WORKFLOW_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_DETRITUS_ENTRY (CALL_CACHING_DETRITUS_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_HASH_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_SIMPLETON_ENTRY (CALL_CACHING_SIMPLETON_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key added to JOB_KEY_VALUE_ENTRY (JOB_KEY_VALUE_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key added to JOB_STORE_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_SIMPLETON_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key added to SUMMARY_STATUS_ENTRY (SUMMARY_STATUS_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key added to WORKFLOW_METADATA_SUMMARY_ENTRY (WORKFLOW_METADATA_SUMMARY_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key added to WORKFLOW_STORE_ENTRY (WORKFLOW_STORE_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_DETRITUS_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_ENTRY.CALL_CACHING_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_SIMPLETON_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to JOB_KEY_VALUE_ENTRY.JOB_KEY_VALUE_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_ENTRY.JOB_STORE_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_SIMPLETON_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to SUMMARY_STATUS_ENTRY.SUMMARY_STATUS_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to WORKFLOW_METADATA_SUMMARY_ENTRY.WORKFLOW_METADATA_SUMMARY_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to WORKFLOW_STORE_ENTRY.WORKFLOW_STORE_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_DETRITUS_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_SIMPLETON_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_DETRITUS_ENTRY(CALL_CACHING_ENTRY_ID, DETRITUS_KEY) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_HASH_ENTRY(CALL_CACHING_ENTRY_ID, HASH_KEY) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_SIMPLETON_ENTRY(CALL_CACHING_ENTRY_ID, SIMPLETON_KEY) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to JOB_KEY_VALUE_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX, JOB_ATTEMPT, STORE_KEY) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX, JOB_ATTEMPT) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE_SIMPLETON_ENTRY(JOB_STORE_ENTRY_ID, SIMPLETON_KEY) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS_ENTRY(SUMMARY_TABLE_NAME, SUMMARIZED_TABLE_NAME) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_METADATA_SUMMARY_ENTRY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_STORE_ENTRY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index IX_JOB_STORE_ENTRY_WEU created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WN created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WS created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index IX_WORKFLOW_STORE_ENTRY_WS created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_again.xml::standardize_column_names_again::kshakir ran successfully in 204ms Running Changeset: changesets/rename_workflow_options_in_metadata.xml::rename_workflow_options_in_metadata::tjeandet [2024-01-10 14:22:49,23] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000 [2024-01-10 14:22:49,25] [info] [RenameWorkflowOptionsInMetadata] 100% Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: RenameWorkflowOptionsInMetadata complete. Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/rename_workflow_options_in_metadata.xml::rename_workflow_options_in_metadata::tjeandet ran successfully in 23ms Running Changeset: changesets/encrypt_and_clear_workflow_options.xml::encrypt_workflow_store_entry_workflow_options::kshakir Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: EncryptWorkflowStoreEntryWorkflowOptions complete. Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/encrypt_and_clear_workflow_options.xml::encrypt_workflow_store_entry_workflow_options::kshakir ran successfully in 3ms Running Changeset: changesets/encrypt_and_clear_workflow_options.xml::clear_metadata_entry_workflow_options::kshakir Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ClearMetadataEntryWorkflowOptions complete. Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/encrypt_and_clear_workflow_options.xml::clear_metadata_entry_workflow_options::kshakir ran successfully in 2ms Running Changeset: changesets/sub_workflow_store.xml::SUB_WORKFLOW_STORE_ENTRY::tjeandet Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table SUB_WORKFLOW_STORE_ENTRY created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::SUB_WORKFLOW_STORE_ENTRY::tjeandet ran successfully in 6ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_uuid_index::tjeandet Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index IX_SUB_WORKFLOW_STORE_ENTRY_PWEU created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_uuid_index::tjeandet ran successfully in 4ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_jobkey_index::tjeandet Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to SUB_WORKFLOW_STORE_ENTRY(PARENT_WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, CALL_INDEX, CALL_ATTEMPT) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_jobkey_index::tjeandet ran successfully in 2ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_root_workflow_fk::tjeandet Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to SUB_WORKFLOW_STORE_ENTRY (ROOT_WORKFLOW_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_root_workflow_fk::tjeandet ran successfully in 2ms Running Changeset: changesets/workflow_store_imports_file.xml::workflow-store-imports-file::cjllanwarne Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Columns IMPORTS_ZIP(BLOB) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_imports_file.xml::workflow-store-imports-file::cjllanwarne ran successfully in 2ms Running Changeset: changesets/workflow_store_labels_file.xml::workflow-store-labels-file::cjllanwarne Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Columns CUSTOM_LABELS(LONGVARCHAR) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_labels_file.xml::workflow-store-labels-file::cjllanwarne ran successfully in 2ms Running Changeset: changesets/embiggen_detritus_value.xml::embiggen_call_caching_detritus_entry::rmunshi Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE datatype was changed to LONGVARCHAR Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/embiggen_detritus_value.xml::embiggen_call_caching_detritus_entry::rmunshi ran successfully in 2ms Running Changeset: changesets/standardize_column_names_patches.xml::standardize_column_names_patches::kshakir Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_HASH_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key FK_JOB_STORE_RESULT_SIMPLETON_JOB_STORE_ENTRY_ID dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key FK_SUB_WORKFLOW_STORE_ROOT_WORKFLOW_ID_WORKFLOW_STORE_ENTRY_ID dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint UC_CALL_CACHING_ENTRY_WEU_CQFN_JI dropped from CALL_CACHING_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint UC_CALL_CACHING_HASH_ENTRY_CCEI dropped from CALL_CACHING_HASH_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_HASH_ENTRY(CALL_CACHING_ENTRY_ID, HASH_KEY) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to SUB_WORKFLOW_STORE_ENTRY (ROOT_WORKFLOW_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_patches.xml::standardize_column_names_patches::kshakir ran successfully in 16ms Running Changeset: changesets/standardize_column_names_patches.xml::hsqldb_longvarchar_to_longtext::kshakir Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE datatype was changed to LONGTEXT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.SIMPLETON_VALUE datatype was changed to LONGTEXT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: JOB_STORE_ENTRY.EXCEPTION_MESSAGE datatype was changed to LONGTEXT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: JOB_STORE_SIMPLETON_ENTRY.SIMPLETON_VALUE datatype was changed to LONGTEXT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.CUSTOM_LABELS datatype was changed to LONGTEXT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_DEFINITION datatype was changed to LONGTEXT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_INPUTS datatype was changed to LONGTEXT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_OPTIONS datatype was changed to LONGTEXT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_patches.xml::hsqldb_longvarchar_to_longtext::kshakir ran successfully in 11ms Running Changeset: changesets/nullable_lobs.xml::nullable_lobs::kshakir Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Null constraint dropped from CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Null constraint dropped from CALL_CACHING_SIMPLETON_ENTRY.SIMPLETON_VALUE Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Null constraint dropped from JOB_STORE_SIMPLETON_ENTRY.SIMPLETON_VALUE Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_DEFINITION Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_INPUTS Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_OPTIONS Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.CUSTOM_LABELS Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/nullable_lobs.xml::nullable_lobs::kshakir ran successfully in 7ms Running Changeset: changesets/add_attempt_in_call_caching_entry.xml::add_attempt_in_call_caching_entry::tjeandet Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Columns JOB_ATTEMPT(INT) added to CALL_CACHING_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/add_attempt_in_call_caching_entry.xml::add_attempt_in_call_caching_entry::tjeandet ran successfully in 2ms Running Changeset: changesets/replace_empty_custom_labels.xml::custom_labels_not_null::rmunshi Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_STORE_ENTRY.CUSTOM_LABELS Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/replace_empty_custom_labels.xml::custom_labels_not_null::rmunshi ran successfully in 2ms Running Changeset: changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry::tjeandet Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table CALL_CACHING_AGGREGATION_ENTRY created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry::tjeandet ran successfully in 2ms Running Changeset: changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry_keys_and_indexes::tjeandet Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_AGGREGATION_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index IX_CALL_CACHING_AGGREGATION_ENTRY_BA_IFA created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry_keys_and_indexes::tjeandet ran successfully in 3ms Running Changeset: changesets/custom_label_entry.xml::custom_LABEL_entry::rmunshi Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::custom_LABEL_entry::rmunshi ran successfully in 2ms Running Changeset: changesets/custom_label_entry.xml::WMSE_workflow_execution_uuid_foreign_key::rmunshi Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to CUSTOM_LABEL_ENTRY (WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::WMSE_workflow_execution_uuid_foreign_key::rmunshi ran successfully in 2ms Running Changeset: changesets/custom_label_entry.xml::custom_label_entry_index::rmunshi Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, CUSTOM_LABEL_VALUE, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::custom_label_entry_index::rmunshi ran successfully in 1ms Running Changeset: changesets/custom_label_entry.xml::modify label key/value data types::rmunshi Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_KEY datatype was changed to VARCHAR(63) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_VALUE datatype was changed to VARCHAR(63) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::modify label key/value data types::rmunshi ran successfully in 3ms Running Changeset: changesets/docker_hash_store.xml::DOCKER_HASH_STORE_ENTRY::tjeandet Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table DOCKER_HASH_STORE_ENTRY created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store.xml::DOCKER_HASH_STORE_ENTRY::tjeandet ran successfully in 2ms Running Changeset: changesets/docker_hash_store.xml::docker_hash_store_uuid_unique_constraint::tjeandet Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to DOCKER_HASH_STORE_ENTRY(WORKFLOW_EXECUTION_UUID, DOCKER_TAG) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store.xml::docker_hash_store_uuid_unique_constraint::tjeandet ran successfully in 3ms Running Changeset: changesets/workflow_store_type_and_version.xml::workflow-store-type-and-version::mcovarr Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Columns WORKFLOW_TYPE(VARCHAR(30)),WORKFLOW_TYPE_VERSION(VARCHAR(255)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_type_and_version.xml::workflow-store-type-and-version::mcovarr ran successfully in 2ms Running Changeset: changesets/remove_pre_pbe_tables.xml::drop-pre-pbe-tables::mcovarr Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table EXECUTION_INFO dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table EXECUTION_EVENT dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table FAILURE_EVENT dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table RUNTIME_ATTRIBUTES dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table SYMBOL dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table EXECUTION dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table WORKFLOW_EXECUTION_AUX dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table WORKFLOW_EXECUTION dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/remove_pre_pbe_tables.xml::drop-pre-pbe-tables::mcovarr ran successfully in 13ms Running Changeset: changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key FK_CUSTOM_LABEL_ENTRY_WORKFLOW_EXECUTION_UUID dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint UC_CUSTOM_LABEL_ENTRY_CLK_CLV_WEU dropped from CUSTOM_LABEL_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint UC_SUMMARY_STATUS_ENTRY_STN_STN dropped from SUMMARY_STATUS_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint UC_WORKFLOW_METADATA_SUMMARY_ENTRY_WEU dropped from WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table METADATA_ENTRY dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table SUMMARY_STATUS_ENTRY dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY_ENTRY dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir ran successfully in 11ms Running Changeset: changesets/workflow_store_state_widening.xml::workflow-store-state-widening::tjeandet Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_STATE datatype was changed to varchar(20) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_state_widening.xml::workflow-store-state-widening::tjeandet ran successfully in 1ms Running Changeset: changesets/workflow_store_restarted_column.xml::workflow-store-restarted-column::tjeandet Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Columns RESTARTED(BOOLEAN) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_restarted_column.xml::workflow-store-restarted-column::tjeandet ran successfully in 4ms Running Changeset: changesets/workflow_store_restarted_column.xml::update-restartable::tjeandet Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Data updated in WORKFLOW_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Data updated in WORKFLOW_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_restarted_column.xml::update-restartable::tjeandet ran successfully in 3ms Running Changeset: changesets/workflow_store_workflow_root_column.xml::workflow-store-workflow-root-column::tjeandet Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Columns WORKFLOW_ROOT(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_workflow_root_column.xml::workflow-store-workflow-root-column::tjeandet ran successfully in 1ms Running Changeset: changesets/workflow_store_horizontal_db.xml::workflow-store-horizontal-db::mcovarr Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column WORKFLOW_STORE_ENTRY.RESTARTED dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Columns CROMWELL_ID(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Columns HEARTBEAT_TIMESTAMP(TIMESTAMP) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_horizontal_db.xml::workflow-store-horizontal-db::mcovarr ran successfully in 5ms Running Changeset: changesets/add_workflow_url_in_workflow_store_entry.xml::add_workflow_url_in_workflow_store_entry::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Columns WORKFLOW_URL(VARCHAR(255)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/add_workflow_url_in_workflow_store_entry.xml::add_workflow_url_in_workflow_store_entry::sshah ran successfully in 2ms Running Changeset: changesets/change_max_size_workflow_url.xml::change_max_size_for_workflow_url::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_URL datatype was changed to VARCHAR(2000) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/change_max_size_workflow_url.xml::change_max_size_for_workflow_url::sshah ran successfully in 2ms Running Changeset: changesets/docker_hash_store_add_size_column.xml::docker_hash_store_entry_add_size_column::tjeandet Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Columns DOCKER_SIZE(BIGINT) added to DOCKER_HASH_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store_add_size_column.xml::docker_hash_store_entry_add_size_column::tjeandet ran successfully in 3ms Running Changeset: changesets/enlarge_call_caching_hash_entry_id.xml::restore_auto_increment_call_caching_hash_entry_id::mcovarr Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_hash_entry_id.xml::restore_auto_increment_call_caching_hash_entry_id::mcovarr ran successfully in 1ms Running Changeset: changesets/add_hog_group_in_workflow_store.xml::add_hog_group_in_workflow_store::cjllanwarne Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Columns HOG_GROUP(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/add_hog_group_in_workflow_store.xml::add_hog_group_in_workflow_store::cjllanwarne ran successfully in 2ms Running Changeset: changesets/resync_engine_schema.xml::resync-engine-schema::kshakir Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.HEARTBEAT_TIMESTAMP datatype was changed to DATETIME(6) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.SUBMISSION_TIME datatype was changed to DATETIME(6) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_STORE_ENTRY.SUBMISSION_TIME Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/resync_engine_schema.xml::resync-engine-schema::kshakir ran successfully in 3ms Running Changeset: changesets/enlarge_job_store_ids.xml::drop_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key FK_JOB_STORE_SIMPLETON_ENTRY_JOB_STORE_ENTRY_ID dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::drop_job_store_simpleton_entry_fk::mcovarr ran successfully in 1ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_entry_id::mcovarr Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_ENTRY.JOB_STORE_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_entry_id::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_id::mcovarr Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_SIMPLETON_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_id::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_fk::mcovarr ran successfully in 3ms Running Changeset: changesets/enlarge_job_store_ids.xml::recreate_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::recreate_job_store_simpleton_entry_fk::mcovarr ran successfully in 3ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_AGGREGATION_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_detritus_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_DETRITUS_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_detritus_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_hash_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_HASH_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_hash_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_SIMPLETON_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_cache_entry_id::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CALL_CACHING_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_cache_entry_id::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::reset_call_cache_entry_id_autoincrement::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_ENTRY.CALL_CACHING_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::reset_call_cache_entry_id_autoincrement::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_aggregation_entry_fk::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_aggregation_entry_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_detritus_entry_fk::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_detritus_entry_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_hash_entry_fk::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CALL_CACHING_HASH_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_hash_entry_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_simpleton_entry_fk::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_simpleton_entry_fk::sshah ran successfully in 3ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_AGGREGATION_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah ran successfully in 3ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_detritus_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_DETRITUS_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_detritus_entry_call_caching_entry_id_fk::sshah ran successfully in 3ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_hash_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_hash_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_SIMPLETON_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_caching_aggregation_entry_id.xml::enlarge_call_cache_aggregation_entry_id::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_AGGREGATION_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_aggregation_entry_id.xml::enlarge_call_cache_aggregation_entry_id::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_aggregation_entry_id.xml::reset_call_caching_aggregation_entry_id_autoincrement::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_AGGREGATION_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_aggregation_entry_id.xml::reset_call_caching_aggregation_entry_id_autoincrement::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_detritus_entry_id.xml::enlarge_call_cache_detritus_entry_id::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_DETRITUS_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_detritus_entry_id.xml::enlarge_call_cache_detritus_entry_id::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_caching_detritus_entry_id.xml::reset_call_caching_detritus_entry_id_autoincrement::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_DETRITUS_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_detritus_entry_id.xml::reset_call_caching_detritus_entry_id_autoincrement::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_caching_simpleton_entry_id.xml::enlarge_call_cache_simpleton_entry_id::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_SIMPLETON_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_simpleton_entry_id.xml::enlarge_call_cache_simpleton_entry_id::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_caching_simpleton_entry_id.xml::reset_call_caching_simpleton_entry_id_autoincrement::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_SIMPLETON_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_simpleton_entry_id.xml::reset_call_caching_simpleton_entry_id_autoincrement::sshah ran successfully in 2ms Running Changeset: changesets/reset_call_caching_hash_entry_id_autoincrement.xml::reset_call_caching_hash_entry_id_autoincrement::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/reset_call_caching_hash_entry_id_autoincrement.xml::reset_call_caching_hash_entry_id_autoincrement::sshah ran successfully in 2ms Jan 10, 2024 2:22:49 PM liquibase.lockservice INFO: Successfully released change log lock [2024-01-10 14:22:49,66] [info] Running with database db.url = jdbc:hsqldb:mem:b027f354-c120-4655-83e4-d06157ae8759;shutdown=false;hsqldb.tx=mvcc Jan 10, 2024 2:22:49 PM liquibase.database INFO: Set default schema name to PUBLIC Jan 10, 2024 2:22:49 PM liquibase.lockservice INFO: Successfully acquired change log lock Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Creating database history table with name: PUBLIC.SQLMETADATADATABASECHANGELOG Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Reading from PUBLIC.SQLMETADATADATABASECHANGELOG Running Changeset: metadata_changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table METADATA_ENTRY created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table SUMMARY_STATUS_ENTRY created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY_ENTRY created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, CUSTOM_LABEL_VALUE, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS_ENTRY(SUMMARY_TABLE_NAME, SUMMARIZED_TABLE_NAME) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_METADATA_SUMMARY_ENTRY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Foreign key constraint added to CUSTOM_LABEL_ENTRY (WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WN created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WS created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index METADATA_JOB_AND_KEY_IDX created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index METADATA_JOB_IDX created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index METADATA_WORKFLOW_IDX created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index SYS_IDX_11226 created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir ran successfully in 25ms Running Changeset: changesets/change_max_size_label_entry.xml::change_max_size_for_key_and_value_in_label::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_KEY datatype was changed to VARCHAR(255) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_VALUE datatype was changed to VARCHAR(255) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet changesets/change_max_size_label_entry.xml::change_max_size_for_key_and_value_in_label::sshah ran successfully in 3ms Running Changeset: metadata_changesets/delete_duplicate_custom_labels.xml::delete_duplicate_custom_labels::kshakir Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Data deleted from CUSTOM_LABEL_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint UC_CUSTOM_LABEL_ENTRY_CLK_CLV_WEU dropped from CUSTOM_LABEL_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/delete_duplicate_custom_labels.xml::delete_duplicate_custom_labels::kshakir ran successfully in 6ms Running Changeset: metadata_changesets/add_submission_timestamp_metadata_summary.xml::add_submission_timestamp_in_metadata_entry::sshah Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Columns SUBMISSION_TIMESTAMP(DATETIME(6)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_submission_timestamp_metadata_summary.xml::add_submission_timestamp_in_metadata_entry::sshah ran successfully in 3ms Running Changeset: metadata_changesets/custom_label_entry_key_value_index.xml::custom_label_entry_key_value_index::mcovarr Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index IX_CUSTOM_LABEL_ENTRY_CLK_CLV created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/custom_label_entry_key_value_index.xml::custom_label_entry_key_value_index::mcovarr ran successfully in 1ms Running Changeset: metadata_changesets/add_parent_and_root_workflow_execution_uuid.xml::add_parent_and_root_workflow_execution_uuid::kshakir Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Columns PARENT_WORKFLOW_EXECUTION_UUID(VARCHAR(100)),ROOT_WORKFLOW_EXECUTION_UUID(VARCHAR(100)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_PWEU created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_RWEU created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint UC_SUMMARY_STATUS_ENTRY_STN_STN dropped from SUMMARY_STATUS_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.SUMMARY_TABLE_NAME renamed to SUMMARY_NAME Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.SUMMARIZED_TABLE_NAME dropped Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.MAXIMUM_ID renamed to SUMMARY_POSITION Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS_ENTRY(SUMMARY_NAME) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_parent_and_root_workflow_execution_uuid.xml::add_parent_and_root_workflow_execution_uuid::kshakir ran successfully in 13ms Running Changeset: metadata_changesets/add_parent_and_root_workflow_typo_fix.xml::add_parent_and_root_workflow_typo_fix::kshakir Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Data deleted from SUMMARY_STATUS_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_parent_and_root_workflow_typo_fix.xml::add_parent_and_root_workflow_typo_fix::kshakir ran successfully in 2ms Running Changeset: metadata_changesets/metadata_index_removals.xml::metadata_index_removals::mcovarr Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index METADATA_JOB_AND_KEY_IDX dropped from table METADATA_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index METADATA_JOB_IDX dropped from table METADATA_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/metadata_index_removals.xml::metadata_index_removals::mcovarr ran successfully in 3ms Running Changeset: metadata_changesets/add_metadata_archive_status.xml::add_metadata_archive_status::cjllanwarne Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Columns METADATA_ARCHIVE_STATUS(VARCHAR(30)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_metadata_archive_status.xml::add_metadata_archive_status::cjllanwarne ran successfully in 2ms Running Changeset: metadata_changesets/add_metadata_archive_status.xml::metadata_archive_status_index::cjllanwarne Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_metadata_archive_status.xml::metadata_archive_status_index::cjllanwarne ran successfully in 1ms Running Changeset: metadata_changesets/summarization_queue_table.xml::summary_queue_table::gsterin Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Table SUMMARY_QUEUE_ENTRY created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table.xml::summary_queue_table::gsterin ran successfully in 2ms Running Changeset: metadata_changesets/summarization_queue_table.xml::summary_queue_initialization::gsterin Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table.xml::summary_queue_initialization::gsterin ran successfully in 6ms Running Changeset: metadata_changesets/summarization_queue_table_add_primary_key.xml::summary_queue_add_primary_key::gsterin Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Primary key added to SUMMARY_QUEUE_ENTRY (METADATA_JOURNAL_ID) Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table_add_primary_key.xml::summary_queue_add_primary_key::gsterin ran successfully in 3ms Running Changeset: metadata_changesets/remove_non_summarizable_metadata_from_queue.xml::delete_non_summarizable_metadata_from_queue::mcovarr Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/remove_non_summarizable_metadata_from_queue.xml::delete_non_summarizable_metadata_from_queue::mcovarr ran successfully in 4ms Running Changeset: metadata_changesets/update_metadata_archive_index.xml::update_old_metadata_archive_status_index::cjllanwarne Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS dropped from table WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS_ET created Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/update_metadata_archive_index.xml::update_old_metadata_archive_status_index::cjllanwarne ran successfully in 2ms Running Changeset: metadata_changesets/reset_archive_statuses_to_null.xml::reset_archive_statuses_to_null::cjllanwarne Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: Data updated in WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:22:49 PM liquibase.changelog INFO: ChangeSet metadata_changesets/reset_archive_statuses_to_null.xml::reset_archive_statuses_to_null::cjllanwarne ran successfully in 2ms Jan 10, 2024 2:22:49 PM liquibase.lockservice INFO: Successfully released change log lock [2024-01-10 14:22:50,30] [info] Slf4jLogger started [2024-01-10 14:22:50,60] [info] Workflow heartbeat configuration: { "cromwellId" : "cromid-cfbf762", "heartbeatInterval" : "2 minutes", "ttl" : "10 minutes", "failureShutdownDuration" : "5 minutes", "writeBatchSize" : 10000, "writeThreshold" : 10000 } [2024-01-10 14:22:50,70] [info] Metadata summary refreshing every 1 second. [2024-01-10 14:22:50,70] [info] No metadata archiver defined in config [2024-01-10 14:22:50,71] [info] No metadata deleter defined in config [2024-01-10 14:22:50,74] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds. [2024-01-10 14:22:50,74] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds. [2024-01-10 14:22:50,79] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds. [2024-01-10 14:22:50,90] [info] JobRestartCheckTokenDispenser - Distribution rate: 50 per 1 seconds. [2024-01-10 14:22:50,94] [info] JobExecutionTokenDispenser - Distribution rate: 20 per 10 seconds. [2024-01-10 14:22:51,00] [info] SingleWorkflowRunnerActor: Version 83 [2024-01-10 14:22:51,01] [info] SingleWorkflowRunnerActor: Submitting workflow [2024-01-10 14:22:51,04] [info] 1 new workflows fetched by cromid-cfbf762: 4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1 [2024-01-10 14:22:51,06] [info] WorkflowManagerActor: Starting workflow 4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1 [2024-01-10 14:22:51,07] [info] Unspecified type (Unspecified version) workflow 4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1 submitted [2024-01-10 14:22:51,09] [info] WorkflowManagerActor: Successfully started WorkflowActor-4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1 [2024-01-10 14:22:51,09] [info] Retrieved 1 workflows from the WorkflowStoreActor [2024-01-10 14:22:51,11] [info] SingleWorkflowRunnerActor: Workflow submitted 4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1 [2024-01-10 14:22:51,14] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes. [2024-01-10 14:22:51,28] [info] MaterializeWorkflowDescriptorActor [4cd7fc7e]: Parsing workflow as WDL 1.0 [2024-01-10 14:22:53,48] [info] MaterializeWorkflowDescriptorActor [4cd7fc7e]: Call-to-Backend assignments: theiacov_illumina_se.pangolin4 -> Local, theiacov_illumina_se.nextclade -> Local, theiacov_illumina_se.raw_check_reads -> Local, ivar_consensus.bwa -> Local, theiacov_illumina_se.vadr -> Local, read_QC_trim_se.bbduk_se -> Local, theiacov_illumina_se.nextclade_output_parser -> Local, read_QC_trim_se.kraken2_raw -> Local, read_QC_trim_se.fastqc_clean -> Local, read_QC_trim_se.kraken2_standalone -> Local, theiacov_illumina_se.qc_check_task -> Local, theiacov_illumina_se.sc2_gene_coverage -> Local, read_QC_trim_se.trimmomatic_se -> Local, read_QC_trim_se.fastp_se -> Local, ivar_consensus.primer_trim -> Local, theiacov_illumina_se.clean_check_reads -> Local, ivar_consensus.stats_n_coverage -> Local, read_QC_trim_se.fastq_scan_clean -> Local, read_QC_trim_se.midas -> Local, ivar_consensus.consensus -> Local, read_QC_trim_se.fastq_scan_raw -> Local, read_QC_trim_se.fastqc_raw -> Local, theiacov_illumina_se.version_capture -> Local, ivar_consensus.stats_n_coverage_primtrim -> Local, ivar_consensus.variant_call -> Local, theiacov_illumina_se.consensus_qc -> Local [2024-01-10 14:22:53,71] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,71] [warn] Local [4cd7fc7e]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,71] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,71] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,71] [warn] Local [4cd7fc7e]: Key/s [dx_instance_type, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,71] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,71] [warn] Local [4cd7fc7e]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,71] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,71] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,72] [warn] Local [4cd7fc7e]: Key/s [preemptible, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,72] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,72] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,72] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,72] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,72] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,72] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,72] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,72] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,72] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,73] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,73] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,73] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,73] [warn] Local [4cd7fc7e]: Key/s [dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,73] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,73] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:53,73] [warn] Local [4cd7fc7e]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:22:54,83] [info] WorkflowExecutionActor-4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1 [4cd7fc7e]: Starting theiacov_illumina_se.version_capture [2024-01-10 14:22:55,90] [info] WorkflowExecutionActor-4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1 [4cd7fc7e]: Starting theiacov_illumina_se.raw_check_reads [2024-01-10 14:22:55,91] [info] Not triggering log of restart checking token queue status. Effective log interval = None [2024-01-10 14:22:55,95] [info] Not triggering log of execution token queue status. Effective log interval = None [2024-01-10 14:23:00,96] [info] Assigned new job execution tokens to the following groups: 4cd7fc7e: 2 [2024-01-10 14:23:02,06] [warn] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.raw_check_reads:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:23:02,07] [warn] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.version_capture:NA:1]: Unrecognized runtime attribute keys: dx_instance_type, disks, cpu, memory [2024-01-10 14:23:02,18] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.version_capture:NA:1]: PHB_Version="PHB v1.2.1" date +"%Y-%m-%d" > TODAY echo "$PHB_Version" > PHB_VERSION [2024-01-10 14:23:02,19] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.raw_check_reads:NA:1]: flag="PASS" # initalize estimated genome size estimated_genome_size=0 if [[ "false" == "false" ]] ; then # set cat command based on compression if [[ "/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-raw_check_reads/inputs/1422045097/ERR6319327.fastq.gz" == *".gz" ]] ; then cat_reads="zcat" else cat_reads="cat" fi # check one: number of reads read1_num=eval "$cat_reads /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-raw_check_reads/inputs/1422045097/ERR6319327.fastq.gz" | awk '{s++}END{print s/4}' # awk '{s++}END{print s/4' counts the number of lines and divides them by 4 # key assumption: in fastq there will be four lines per read # sometimes fastqs do not have 4 lines per read, so this might fail one day if [ "${read1_num}" -le "113" ] ; then flag="FAIL; the number of reads (${read1_num}) is below the minimum of 113" else flag="PASS" fi # checks two and three: number of basepairs and proportion of sequence if [ "${flag}" == "PASS" ]; then # count number of basepairs # this only works if the fastq has 4 lines per read, so this might fail one day read1_bp=eval "${cat_reads} /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-raw_check_reads/inputs/1422045097/ERR6319327.fastq.gz" | paste - - - - | cut -f2 | tr -d '\n' | wc -c # paste - - - - (print 4 consecutive lines in one row, tab delimited) # cut -f2 print only the second column (the second line of the fastq 4-line) # tr -d '\n' removes line endings # wc -c counts characters if [ "$flag" == "PASS" ] ; then if [ "${read1_bp}" -le "34000" ] ; then flag="FAIL; the number of basepairs (${read1_bp}) is below the minimum of 34000" else flag="PASS" fi fi fi #checks four and five: estimated genome size and coverage if [ "${flag}" == "PASS" ] && [ "false" == "false" ]; then # estimate genome size if theiaprok AND expected_genome_size was not provided if [ "theiacov" == "theiaprok" ] && [[ -z "" ]]; then # First Pass; assuming average depth mash sketch -o test -k 31 -m 3 -r /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-raw_check_reads/inputs/1422045097/ERR6319327.fastq.gz > mash-output.txt 2>&1 grep "Estimated genome size:" mash-output.txt | \ awk '{if($4){printf("%d", $4)}} END {if (!NR) print "0"}' > genome_size_output grep "Estimated coverage:" mash-output.txt | \ awk '{if($3){printf("%d", $3)}} END {if (!NR) print "0"}' > coverage_output # remove mash outputs rm -rf test.msh rm -rf mash-output.txt estimated_genome_size=head -n1 genome_size_output estimated_coverage=head -n1 coverage_output # Check if second pass is needed if [ ${estimated_genome_size} -gt "2673870" ] || [ ${estimated_genome_size} -lt "1700" ] ; then # Probably high coverage, try increasing number of kmer copies to 10 M="-m 10" if [ ${estimated_genome_size} -lt "1700" ]; then # Probably low coverage, try decreasing the number of kmer copies to 1 M="-m 1" fi mash sketch -o test -k 31 ${M} -r /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-raw_check_reads/inputs/1422045097/ERR6319327.fastq.gz > mash-output.txt 2>&1 grep "Estimated genome size:" mash-output.txt | \ awk '{if($4){printf("%d", $4)}} END {if (!NR) print "0"}' > genome_size_output grep "Estimated coverage:" mash-output.txt | \ awk '{if($3){printf("%d", $3)}} END {if (!NR) print "0"}' > coverage_output # remove mash outputs rm -rf test.msh rm -rf mash-output.txt fi estimated_genome_size=head -n1 genome_size_output estimated_coverage=head -n1 coverage_output # estimate coverage if theiacov OR expected_genome_size was provided elif [ "theiacov" == "theiacov" ] || [ "" ]; then if [ "" ]; then estimated_genome_size= # use user-provided expected_genome_size elif [ "sars-cov-2" == "sars-cov-2" ]; then estimated_genome_size=29903 # size taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_045512.2 elif [ "sars-cov-2" == "MPXV" ]; then estimated_genome_size=197209 # size of 2022 virus taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_063383.1 elif [ "sars-cov-2" == "flu" ]; then estimated_genome_size=14000 # 500 bp over the CDC's approximate full genome size of 13500 (see https://www.cdc.gov/flu/about/professionals/genetic-characterization.htm) elif [ "sars-cov-2" == "HIV" ]; then estimated_genome_size=9181 # size taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_001802.1 elif [ "sars-cov-2" == "WNV" ]; then estimated_genome_size=11092 # WNV lineage 1 size from https://www.ncbi.nlm.nih.gov/nuccore/NC_009942.1 else flag="FAIL; the organism tag provided (sars-cov-2) is not valid and no expected_genome_size was provided." fi # coverage is calculated here by N/G where N is number of bases, and G is genome length # this will nearly always be an overestimation if [ $estimated_genome_size -ne 0 ]; then # prevent divided by zero errors estimated_coverage=$(python3 -c "print(round(($read1_bp)/$estimated_genome_size))") else # they provided 0 for estimated_genome_size, nice estimated_coverage=0 fi else # workflow series was not provided; default to fail estimated_genome_size=0 estimated_coverage=0 fi if [ "${estimated_genome_size}" -ge "2673870" ] ; then flag="FAIL; the estimated genome size (${estimated_genome_size}) is larger than the maximum of 2673870 bps" elif [ "${estimated_genome_size}" -le "1700" ] ; then flag="FAIL; the estimated genome size (${estimated_genome_size}) is smaller than the minimum of 1700 bps" else flag="PASS" if [ "${estimated_coverage}" -lt "10" ] ; then flag="FAIL; the estimated coverage (${estimated_coverage}) is less than the minimum of 10x" else flag="PASS" echo $estimated_genome_size | tee EST_GENOME_LENGTH fi fi fi fi echo $flag | tee FLAG echo $estimated_genome_size | tee EST_GENOME_LENGTH [2024-01-10 14:23:02,37] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.raw_check_reads:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-raw_check_reads/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-raw_check_reads/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-raw_check_reads:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-raw_check_reads:delegated \ us-docker.pkg.dev/general-theiagen/bactopia/gather_samples@sha256:2e044c7be68519b9d683cd242177518e6db662b40b3bf669f891954f90a4d9a5 /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-raw_check_reads/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-raw_check_reads/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-raw_check_reads/execution/docker_cid # return exit code exit $rc [2024-01-10 14:23:02,37] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.version_capture:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-version_capture/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-version_capture/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-version_capture:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-version_capture:delegated \ us-docker.pkg.dev/general-theiagen/ubuntu/ubuntu@sha256:9a35cef02b57290a7c919824b632897fbe6db3aa18c930110b04d81fcad458ff /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-version_capture/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-version_capture/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-version_capture/execution/docker_cid # return exit code exit $rc [2024-01-10 14:23:05,79] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.raw_check_reads:NA:1]: job id: 56165 [2024-01-10 14:23:05,79] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.version_capture:NA:1]: job id: 56164 [2024-01-10 14:23:05,79] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.version_capture:NA:1]: Status change from - to Done [2024-01-10 14:23:05,79] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.raw_check_reads:NA:1]: Status change from - to Done [2024-01-10 14:23:13,27] [info] 9a62bb54-35ab-4405-a3da-d6a8089dca27-SubWorkflowActor-SubWorkflow-read_QC_trim:-1:1 [9a62bb54]: Starting read_QC_trim_se.trimmomatic_se, read_QC_trim_se.fastq_scan_raw [2024-01-10 14:23:20,96] [info] Assigned new job execution tokens to the following groups: 4cd7fc7e: 2 [2024-01-10 14:23:21,17] [warn] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.fastq_scan_raw:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:23:21,18] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.fastq_scan_raw:NA:1]: # capture date and version date | tee DATE fastq-scan -v | tee VERSION # set cat command based on compression if [[ "/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_raw/inputs/1422045097/ERR6319327.fastq.gz" == *".gz" ]] ; then cat_reads="zcat" else cat_reads="cat" fi # capture forward read stats eval "${cat_reads} /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_raw/inputs/1422045097/ERR6319327.fastq.gz" | fastq-scan | tee ERR6319327_fastq-scan.json cat ERR6319327_fastq-scan.json | jq .qc_stats.read_total | tee READ1_SEQS [2024-01-10 14:23:21,20] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.fastq_scan_raw:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_raw/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_raw/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_raw:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_raw:delegated \ us-docker.pkg.dev/general-theiagen/biocontainers/fastq-scan@sha256:dd4880ca592ef724a15cd63dfcb4e3e04f5d4b269976e563ab7413d8ca5cfcfa /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_raw/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_raw/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_raw/execution/docker_cid # return exit code exit $rc [2024-01-10 14:23:21,21] [warn] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.trimmomatic_se:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:23:21,22] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.trimmomatic_se:NA:1]: # date and version control date | tee DATE trimmomatic -version > VERSION && sed -i -e 's/^/Trimmomatic /' VERSION trimmomatic SE \ \ -threads 4 \ /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-trimmomatic_se/inputs/1422045097/ERR6319327.fastq.gz \ ERR6319327_trimmed.fastq.gz \ SLIDINGWINDOW:4:30 \ MINLEN:25 > ERR6319327.trim.stats.txt [2024-01-10 14:23:21,24] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.trimmomatic_se:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-trimmomatic_se/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-trimmomatic_se/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-trimmomatic_se:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-trimmomatic_se:delegated \ us-docker.pkg.dev/general-theiagen/staphb/trimmomatic@sha256:57b673e66313e355a447e4fa1a78fd3ba1ae3ddd8c8f91358efe99140acb5ddb /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-trimmomatic_se/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-trimmomatic_se/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-trimmomatic_se/execution/docker_cid # return exit code exit $rc [2024-01-10 14:23:25,77] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.trimmomatic_se:NA:1]: job id: 56588 [2024-01-10 14:23:25,77] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.fastq_scan_raw:NA:1]: job id: 56577 [2024-01-10 14:23:25,77] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.trimmomatic_se:NA:1]: Status change from - to Done [2024-01-10 14:23:25,77] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.fastq_scan_raw:NA:1]: Status change from - to Done [2024-01-10 14:23:28,56] [info] 9a62bb54-35ab-4405-a3da-d6a8089dca27-SubWorkflowActor-SubWorkflow-read_QC_trim:-1:1 [9a62bb54]: Starting read_QC_trim_se.bbduk_se [2024-01-10 14:23:30,95] [info] Assigned new job execution tokens to the following groups: 4cd7fc7e: 1 [2024-01-10 14:23:31,15] [warn] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.bbduk_se:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:23:31,15] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se/inputs/1475267396/ERR6319327_trimmed.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-trimmomatic_se/execution/ERR6319327_trimmed.fastq.gz: Operation not permitted [2024-01-10 14:23:31,17] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.bbduk_se:NA:1]: # date and version control date | tee DATE # set adapter fasta if [[ ! -z "" ]]; then echo "Using user supplied FASTA file for adapters..." adapter_fasta="" else echo "User did not supply adapters FASTA file, using default adapters.fa file..." adapter_fasta="/bbmap/resources/adapters.fa" fi # set phix fasta if [[ ! -z "" ]]; then echo "Using user supplied FASTA file for phiX..." phix_fasta="" else echo "User did not supply phiX FASTA file, using default phix174_ill.ref.fa.gz file..." phix_fasta="/bbmap/resources/phix174_ill.ref.fa.gz" fi bbduk.sh in1=/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se/inputs/1475267396/ERR6319327_trimmed.fastq.gz out1=ERR6319327.rmadpt_1.fastq.gz ref=${adapter_fasta} stats=ERR6319327.adapters.stats.txt ktrim=r k=23 mink=11 hdist=1 tpe tbo ordered=t bbduk.sh in1=/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se/inputs/1475267396/ERR6319327_trimmed.fastq.gz out1=ERR6319327_1.clean.fastq.gz outm=ERR6319327.matched_phix.fq ref=${phix_fasta} k=31 hdist=1 stats=ERR6319327.phix.stats.txt ordered=t [2024-01-10 14:23:31,18] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.bbduk_se:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se:delegated \ us-docker.pkg.dev/general-theiagen/staphb/bbtools@sha256:fe632b5aa277b939efcf9cdfdd65703133ba3c01ec50db2833b2bb3cf7048ec2 /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se/execution/docker_cid # return exit code exit $rc [2024-01-10 14:23:35,76] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.bbduk_se:NA:1]: job id: 56989 [2024-01-10 14:23:35,77] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.bbduk_se:NA:1]: Status change from - to Done [2024-01-10 14:23:37,74] [info] 9a62bb54-35ab-4405-a3da-d6a8089dca27-SubWorkflowActor-SubWorkflow-read_QC_trim:-1:1 [9a62bb54]: Starting read_QC_trim_se.fastq_scan_clean, read_QC_trim_se.kraken2_raw [2024-01-10 14:23:40,95] [info] Assigned new job execution tokens to the following groups: 4cd7fc7e: 2 [2024-01-10 14:23:40,98] [warn] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.fastq_scan_clean:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:23:40,99] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_clean/inputs/-749931584/ERR6319327_1.clean.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se/execution/ERR6319327_1.clean.fastq.gz: Operation not permitted [2024-01-10 14:23:41,03] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.fastq_scan_clean:NA:1]: # capture date and version date | tee DATE fastq-scan -v | tee VERSION # set cat command based on compression if [[ "/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_clean/inputs/-749931584/ERR6319327_1.clean.fastq.gz" == *".gz" ]] ; then cat_reads="zcat" else cat_reads="cat" fi # capture forward read stats eval "${cat_reads} /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_clean/inputs/-749931584/ERR6319327_1.clean.fastq.gz" | fastq-scan | tee ERR6319327_1.clean_fastq-scan.json cat ERR6319327_1.clean_fastq-scan.json | jq .qc_stats.read_total | tee READ1_SEQS [2024-01-10 14:23:41,08] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.fastq_scan_clean:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_clean/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_clean/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_clean:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_clean:delegated \ us-docker.pkg.dev/general-theiagen/biocontainers/fastq-scan@sha256:dd4880ca592ef724a15cd63dfcb4e3e04f5d4b269976e563ab7413d8ca5cfcfa /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_clean/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_clean/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-fastq_scan_clean/execution/docker_cid # return exit code exit $rc [2024-01-10 14:23:41,18] [warn] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.kraken2_raw:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:23:41,20] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-kraken2_raw/inputs/-749931584/ERR6319327_1.clean.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se/execution/ERR6319327_1.clean.fastq.gz: Operation not permitted [2024-01-10 14:23:41,23] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.kraken2_raw:NA:1]: # date and version control date | tee DATE kraken2 --version | head -n1 | tee VERSION num_reads=$(ls *fastq.gz 2> /dev/nul | wc -l) if ! [ -z ]; then mode="--paired" fi echo $mode kraken2 $mode \ --threads 4 \ --db /kraken2-db \ /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-kraken2_raw/inputs/-749931584/ERR6319327_1.clean.fastq.gz \ --report ERR6319327_kraken2_report.txt >/dev/null percentage_human=$(grep "Homo sapiens" ERR6319327_kraken2_report.txt | cut -f 1) # | tee PERCENT_HUMAN percentage_sc2=$(grep "Severe acute respiratory syndrome coronavirus 2" ERR6319327_kraken2_report.txt | cut -f1 ) # | tee PERCENT_COV if [ -z "$percentage_human" ] ; then percentage_human="0" ; fi if [ -z "$percentage_sc2" ] ; then percentage_sc2="0" ; fi echo $percentage_human | tee PERCENT_HUMAN echo $percentage_sc2 | tee PERCENT_SC2 # capture target org percentage if [ ! -z "" ]; then echo "Target org designated: " percent_target_org=$(grep "" ERR6319327_kraken2_report.txt | cut -f1 | head -n1 ) if [-z "$percent_target_org" ] ; then percent_target_org="0" ; fi else percent_target_org="" fi echo $percent_target_org | tee PERCENT_TARGET_ORG [2024-01-10 14:23:41,42] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.kraken2_raw:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-kraken2_raw/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-kraken2_raw/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-kraken2_raw:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-kraken2_raw:delegated \ us-docker.pkg.dev/general-theiagen/staphb/kraken2@sha256:7884b5a6ec3c20cd7edbd42b92ae7e9806b4581997acb270b1b9ecb627474e2a /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-kraken2_raw/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-kraken2_raw/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-kraken2_raw/execution/docker_cid # return exit code exit $rc [2024-01-10 14:23:45,76] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.fastq_scan_clean:NA:1]: job id: 57332 [2024-01-10 14:23:45,77] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.kraken2_raw:NA:1]: job id: 57407 [2024-01-10 14:23:45,77] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.fastq_scan_clean:NA:1]: Status change from - to Done [2024-01-10 14:23:45,77] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.kraken2_raw:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:24:06,43] [info] BackgroundConfigAsyncJobExecutionActor [9a62bb54read_QC_trim_se.kraken2_raw:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:24:09,34] [info] 9a62bb54-35ab-4405-a3da-d6a8089dca27-SubWorkflowActor-SubWorkflow-read_QC_trim:-1:1 [9a62bb54]: Workflow read_QC_trim_se complete. Final Outputs: { "read_QC_trim_se.midas_docker": null, "read_QC_trim_se.fastqc_raw_number_reads": null, "read_QC_trim_se.kraken_sc2": 95.39, "read_QC_trim_se.trimmomatic_version": "Trimmomatic 0.39", "read_QC_trim_se.midas_report": null, "read_QC_trim_se.midas_primary_genus": null, "read_QC_trim_se.fastqc_version": null, "read_QC_trim_se.fastp_version": null, "read_QC_trim_se.fastq_scan_version": "fastq-scan 0.4.4", "read_QC_trim_se.fastqc_docker": null, "read_QC_trim_se.kraken_human": 0.01, "read_QC_trim_se.kraken_target_org": "", "read_QC_trim_se.kraken_docker": "us-docker.pkg.dev/general-theiagen/staphb/kraken2:2.0.8-beta_hv", "read_QC_trim_se.read1_clean": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se/execution/ERR6319327_1.clean.fastq.gz", "read_QC_trim_se.fastqc_clean_number_reads": null, "read_QC_trim_se.bbduk_docker": "us-docker.pkg.dev/general-theiagen/staphb/bbtools:38.76", "read_QC_trim_se.kraken_target_org_name": null, "read_QC_trim_se.midas_secondary_genus_abundance": null, "read_QC_trim_se.kraken_version": "Kraken version 2.0.8-beta", "read_QC_trim_se.fastq_scan_clean_number_reads": 267294, "read_QC_trim_se.fastqc_raw_html": null, "read_QC_trim_se.fastqc_clean_html": null, "read_QC_trim_se.fastq_scan_docker": "quay.io/biocontainers/fastq-scan:0.4.4--h7d875b9_1", "read_QC_trim_se.fastq_scan_raw_number_reads": 270292, "read_QC_trim_se.kraken_report": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-kraken2_raw/execution/ERR6319327_kraken2_report.txt", "read_QC_trim_se.midas_secondary_genus": null } [2024-01-10 14:24:12,36] [info] WorkflowExecutionActor-4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1 [4cd7fc7e]: Starting theiacov_illumina_se.clean_check_reads [2024-01-10 14:24:20,96] [info] Assigned new job execution tokens to the following groups: 4cd7fc7e: 1 [2024-01-10 14:24:20,98] [warn] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.clean_check_reads:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:24:20,99] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-clean_check_reads/inputs/-749931584/ERR6319327_1.clean.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se/execution/ERR6319327_1.clean.fastq.gz: Operation not permitted [2024-01-10 14:24:21,03] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.clean_check_reads:NA:1]: flag="PASS" # initalize estimated genome size estimated_genome_size=0 if [[ "false" == "false" ]] ; then # set cat command based on compression if [[ "/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-clean_check_reads/inputs/-749931584/ERR6319327_1.clean.fastq.gz" == *".gz" ]] ; then cat_reads="zcat" else cat_reads="cat" fi # check one: number of reads read1_num=eval "$cat_reads /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-clean_check_reads/inputs/-749931584/ERR6319327_1.clean.fastq.gz" | awk '{s++}END{print s/4}' # awk '{s++}END{print s/4' counts the number of lines and divides them by 4 # key assumption: in fastq there will be four lines per read # sometimes fastqs do not have 4 lines per read, so this might fail one day if [ "${read1_num}" -le "113" ] ; then flag="FAIL; the number of reads (${read1_num}) is below the minimum of 113" else flag="PASS" fi # checks two and three: number of basepairs and proportion of sequence if [ "${flag}" == "PASS" ]; then # count number of basepairs # this only works if the fastq has 4 lines per read, so this might fail one day read1_bp=eval "${cat_reads} /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-clean_check_reads/inputs/-749931584/ERR6319327_1.clean.fastq.gz" | paste - - - - | cut -f2 | tr -d '\n' | wc -c # paste - - - - (print 4 consecutive lines in one row, tab delimited) # cut -f2 print only the second column (the second line of the fastq 4-line) # tr -d '\n' removes line endings # wc -c counts characters if [ "$flag" == "PASS" ] ; then if [ "${read1_bp}" -le "34000" ] ; then flag="FAIL; the number of basepairs (${read1_bp}) is below the minimum of 34000" else flag="PASS" fi fi fi #checks four and five: estimated genome size and coverage if [ "${flag}" == "PASS" ] && [ "false" == "false" ]; then # estimate genome size if theiaprok AND expected_genome_size was not provided if [ "theiacov" == "theiaprok" ] && [[ -z "" ]]; then # First Pass; assuming average depth mash sketch -o test -k 31 -m 3 -r /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-clean_check_reads/inputs/-749931584/ERR6319327_1.clean.fastq.gz > mash-output.txt 2>&1 grep "Estimated genome size:" mash-output.txt | \ awk '{if($4){printf("%d", $4)}} END {if (!NR) print "0"}' > genome_size_output grep "Estimated coverage:" mash-output.txt | \ awk '{if($3){printf("%d", $3)}} END {if (!NR) print "0"}' > coverage_output # remove mash outputs rm -rf test.msh rm -rf mash-output.txt estimated_genome_size=head -n1 genome_size_output estimated_coverage=head -n1 coverage_output # Check if second pass is needed if [ ${estimated_genome_size} -gt "2673870" ] || [ ${estimated_genome_size} -lt "1700" ] ; then # Probably high coverage, try increasing number of kmer copies to 10 M="-m 10" if [ ${estimated_genome_size} -lt "1700" ]; then # Probably low coverage, try decreasing the number of kmer copies to 1 M="-m 1" fi mash sketch -o test -k 31 ${M} -r /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-clean_check_reads/inputs/-749931584/ERR6319327_1.clean.fastq.gz > mash-output.txt 2>&1 grep "Estimated genome size:" mash-output.txt | \ awk '{if($4){printf("%d", $4)}} END {if (!NR) print "0"}' > genome_size_output grep "Estimated coverage:" mash-output.txt | \ awk '{if($3){printf("%d", $3)}} END {if (!NR) print "0"}' > coverage_output # remove mash outputs rm -rf test.msh rm -rf mash-output.txt fi estimated_genome_size=head -n1 genome_size_output estimated_coverage=head -n1 coverage_output # estimate coverage if theiacov OR expected_genome_size was provided elif [ "theiacov" == "theiacov" ] || [ "" ]; then if [ "" ]; then estimated_genome_size= # use user-provided expected_genome_size elif [ "sars-cov-2" == "sars-cov-2" ]; then estimated_genome_size=29903 # size taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_045512.2 elif [ "sars-cov-2" == "MPXV" ]; then estimated_genome_size=197209 # size of 2022 virus taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_063383.1 elif [ "sars-cov-2" == "flu" ]; then estimated_genome_size=14000 # 500 bp over the CDC's approximate full genome size of 13500 (see https://www.cdc.gov/flu/about/professionals/genetic-characterization.htm) elif [ "sars-cov-2" == "HIV" ]; then estimated_genome_size=9181 # size taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_001802.1 elif [ "sars-cov-2" == "WNV" ]; then estimated_genome_size=11092 # WNV lineage 1 size from https://www.ncbi.nlm.nih.gov/nuccore/NC_009942.1 else flag="FAIL; the organism tag provided (sars-cov-2) is not valid and no expected_genome_size was provided." fi # coverage is calculated here by N/G where N is number of bases, and G is genome length # this will nearly always be an overestimation if [ $estimated_genome_size -ne 0 ]; then # prevent divided by zero errors estimated_coverage=$(python3 -c "print(round(($read1_bp)/$estimated_genome_size))") else # they provided 0 for estimated_genome_size, nice estimated_coverage=0 fi else # workflow series was not provided; default to fail estimated_genome_size=0 estimated_coverage=0 fi if [ "${estimated_genome_size}" -ge "2673870" ] ; then flag="FAIL; the estimated genome size (${estimated_genome_size}) is larger than the maximum of 2673870 bps" elif [ "${estimated_genome_size}" -le "1700" ] ; then flag="FAIL; the estimated genome size (${estimated_genome_size}) is smaller than the minimum of 1700 bps" else flag="PASS" if [ "${estimated_coverage}" -lt "10" ] ; then flag="FAIL; the estimated coverage (${estimated_coverage}) is less than the minimum of 10x" else flag="PASS" echo $estimated_genome_size | tee EST_GENOME_LENGTH fi fi fi fi echo $flag | tee FLAG echo $estimated_genome_size | tee EST_GENOME_LENGTH [2024-01-10 14:24:21,06] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.clean_check_reads:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-clean_check_reads/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-clean_check_reads/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-clean_check_reads:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-clean_check_reads:delegated \ us-docker.pkg.dev/general-theiagen/bactopia/gather_samples@sha256:2e044c7be68519b9d683cd242177518e6db662b40b3bf669f891954f90a4d9a5 /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-clean_check_reads/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-clean_check_reads/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-clean_check_reads/execution/docker_cid # return exit code exit $rc [2024-01-10 14:24:25,76] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.clean_check_reads:NA:1]: job id: 57804 [2024-01-10 14:24:25,77] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.clean_check_reads:NA:1]: Status change from - to Done [2024-01-10 14:24:31,75] [info] 3700ee04-0c79-46e9-80b1-b086abe16c34-SubWorkflowActor-SubWorkflow-ivar_consensus:-1:1 [3700ee04]: Starting ivar_consensus.bwa [2024-01-10 14:24:40,95] [info] Assigned new job execution tokens to the following groups: 4cd7fc7e: 1 [2024-01-10 14:24:41,15] [warn] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.bwa:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:24:41,17] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-bwa/inputs/-749931584/ERR6319327_1.clean.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-read_QC_trim/read_QC_trim_se/9a62bb54-35ab-4405-a3da-d6a8089dca27/call-bbduk_se/execution/ERR6319327_1.clean.fastq.gz: Operation not permitted [2024-01-10 14:24:41,24] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.bwa:NA:1]: # date and version control date | tee DATE echo "BWA $(bwa 2>&1 | grep Version )" | tee BWA_VERSION samtools --version | head -n1 | tee SAMTOOLS_VERSION # set reference genome if [[ ! -z "" ]]; then echo "User reference identified; will be utilized for alignement" ref_genome="" bwa index "" # move to primer_schemes dir; bwa fails if reference file not in this location else ref_genome="/artic-ncov2019/primer_schemes/nCoV-2019/V3/nCoV-2019.reference.fasta" fi # Map with BWA MEM echo "Running bwa mem -t 6 ${ref_genome} /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-bwa/inputs/-749931584/ERR6319327_1.clean.fastq.gz | samtools sort | samtools view -F 4 -o ERR6319327.sorted.bam " bwa mem \ -t 6 \ "${ref_genome}" \ /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-bwa/inputs/-749931584/ERR6319327_1.clean.fastq.gz |\ samtools sort | samtools view -F 4 -o ERR6319327.sorted.bam if [[ ! -z "" ]]; then echo "processing paired reads" samtools fastq -F4 -1 ERR6319327_R1.fastq.gz -2 ERR6319327_R2.fastq.gz ERR6319327.sorted.bam else echo "processing single-end reads" samtools fastq -F4 ERR6319327.sorted.bam | gzip > ERR6319327_R1.fastq.gz fi # index BAMs samtools index ERR6319327.sorted.bam [2024-01-10 14:24:41,26] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.bwa:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-bwa/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-bwa/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-bwa:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-bwa:delegated \ us-docker.pkg.dev/general-theiagen/staphb/ivar@sha256:3c3cb44b0db91912aa5b895c2bd9b7af71feebbae8dc959f0466c21b3d61c40b /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-bwa/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-bwa/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-bwa/execution/docker_cid # return exit code exit $rc [2024-01-10 14:24:45,76] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.bwa:NA:1]: job id: 58058 [2024-01-10 14:24:45,76] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.bwa:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:24:52,45] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.bwa:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:24:54,17] [info] 3700ee04-0c79-46e9-80b1-b086abe16c34-SubWorkflowActor-SubWorkflow-ivar_consensus:-1:1 [3700ee04]: Starting ivar_consensus.primer_trim, ivar_consensus.stats_n_coverage [2024-01-10 14:25:00,94] [info] Assigned new job execution tokens to the following groups: 4cd7fc7e: 2 [2024-01-10 14:25:00,95] [warn] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.primer_trim:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:25:00,96] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim/inputs/-2092030806/ERR6319327.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-bwa/execution/ERR6319327.sorted.bam: Operation not permitted [2024-01-10 14:25:00,98] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.primer_trim:NA:1]: # date and version control echo "artic-v3.primers.bed" | tee PRIMER_NAME date | tee DATE ivar version | head -n1 | tee IVAR_VERSION samtools --version | head -n1 | tee SAMTOOLS_VERSION # trimming primers ivar trim \ -e \ -i /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim/inputs/-2092030806/ERR6319327.sorted.bam \ -b /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim/inputs/1063836572/artic-v3.primers.bed \ -p ERR6319327.primertrim | tee IVAR_OUT # sorting and indexing the trimmed bams samtools sort \ ERR6319327.primertrim.bam \ -o ERR6319327.primertrim.sorted.bam samtools index ERR6319327.primertrim.sorted.bam PCT=$(grep "Trimmed primers from" IVAR_OUT | perl -lape 's/Trimmed primers from (\S+)%.*/$1/') echo $PCT if [[ $PCT = -* ]]; then echo 0; else echo $PCT; fi > IVAR_TRIM_PCT [2024-01-10 14:25:01,00] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.primer_trim:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim:delegated \ us-docker.pkg.dev/general-theiagen/staphb/ivar@sha256:3c3cb44b0db91912aa5b895c2bd9b7af71feebbae8dc959f0466c21b3d61c40b /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim/execution/docker_cid # return exit code exit $rc [2024-01-10 14:25:01,09] [warn] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.stats_n_coverage:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:25:01,09] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage/inputs/-2092030806/ERR6319327.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-bwa/execution/ERR6319327.sorted.bam: Operation not permitted [2024-01-10 14:25:01,11] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.stats_n_coverage:NA:1]: date | tee DATE samtools --version | head -n1 | tee VERSION samtools stats /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage/inputs/-2092030806/ERR6319327.sorted.bam > ERR6319327.stats.txt samtools coverage /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage/inputs/-2092030806/ERR6319327.sorted.bam -m -o ERR6319327.cov.hist samtools coverage /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage/inputs/-2092030806/ERR6319327.sorted.bam -o ERR6319327.cov.txt samtools flagstat /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage/inputs/-2092030806/ERR6319327.sorted.bam > ERR6319327.flagstat.txt coverage=$(cut -f 6 ERR6319327.cov.txt | tail -n 1) depth=$(cut -f 7 ERR6319327.cov.txt | tail -n 1) meanbaseq=$(cut -f 8 ERR6319327.cov.txt | tail -n 1) meanmapq=$(cut -f 9 ERR6319327.cov.txt | tail -n 1) if [ -z "$coverage" ] ; then coverage="0" ; fi if [ -z "$depth" ] ; then depth="0" ; fi if [ -z "$meanbaseq" ] ; then meanbaseq="0" ; fi if [ -z "$meanmapq" ] ; then meanmapq="0" ; fi echo $coverage | tee COVERAGE echo $depth | tee DEPTH echo $meanbaseq | tee MEANBASEQ echo $meanmapq | tee MEANMAPQ [2024-01-10 14:25:01,14] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.stats_n_coverage:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage:delegated \ us-docker.pkg.dev/general-theiagen/staphb/samtools@sha256:ef92cb5e490c75e66e89ab0a113865839e6954435d363af400eaa27177ff7c91 /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage/execution/docker_cid # return exit code exit $rc [2024-01-10 14:25:05,76] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.stats_n_coverage:NA:1]: job id: 58367 [2024-01-10 14:25:05,76] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.primer_trim:NA:1]: job id: 58354 [2024-01-10 14:25:05,77] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.stats_n_coverage:NA:1]: Status change from - to Done [2024-01-10 14:25:05,77] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.primer_trim:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:25:53,71] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.primer_trim:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:25:55,27] [info] 3700ee04-0c79-46e9-80b1-b086abe16c34-SubWorkflowActor-SubWorkflow-ivar_consensus:-1:1 [3700ee04]: Starting ivar_consensus.stats_n_coverage_primtrim [2024-01-10 14:25:56,29] [info] 3700ee04-0c79-46e9-80b1-b086abe16c34-SubWorkflowActor-SubWorkflow-ivar_consensus:-1:1 [3700ee04]: Starting ivar_consensus.consensus, ivar_consensus.variant_call [2024-01-10 14:26:00,96] [info] Assigned new job execution tokens to the following groups: 4cd7fc7e: 3 [2024-01-10 14:26:00,97] [warn] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.stats_n_coverage_primtrim:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:26:00,97] [warn] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.variant_call:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:26:00,97] [warn] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.consensus:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:26:00,98] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-variant_call/inputs/1660897676/ERR6319327.primertrim.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim/execution/ERR6319327.primertrim.sorted.bam: Operation not permitted [2024-01-10 14:26:00,98] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage_primtrim/inputs/1660897676/ERR6319327.primertrim.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim/execution/ERR6319327.primertrim.sorted.bam: Operation not permitted [2024-01-10 14:26:00,99] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-consensus/inputs/1660897676/ERR6319327.primertrim.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim/execution/ERR6319327.primertrim.sorted.bam: Operation not permitted [2024-01-10 14:26:00,99] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.variant_call:NA:1]: # date and version control date | tee DATE ivar version | head -n1 | tee IVAR_VERSION samtools --version | head -n1 | tee SAMTOOLS_VERSION # set reference genome if [[ ! -z "" ]]; then echo "User reference identified; will be utilized for alignement" ref_genome="" bwa index "" # move to primer_schemes dir; bwa fails if reference file not in this location else ref_genome="/artic-ncov2019/primer_schemes/nCoV-2019/V3/nCoV-2019.reference.fasta" fi # set reference gff if [[ ! -z "" ]]; then echo "User reference identified; will be utilized for alignement" ref_gff="" # move to primer_schemes dir; bwa fails if reference file not in this location else ref_gff="/reference/GCF_009858895.2_ASM985889v3_genomic.gff" fi # call variants samtools mpileup \ -A \ -d 600000 \ -B \ -Q 0 \ --reference ${ref_genome} \ /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-variant_call/inputs/1660897676/ERR6319327.primertrim.sorted.bam | \ ivar variants \ -p ERR6319327.variants \ -q 20 \ -t 0.6 \ -m 100 \ -r ${ref_genome} \ -g ${ref_gff} # Convert TSV to VCF ivar_variants_to_vcf.py ERR6319327.variants.tsv ERR6319327.variants.vcf # Variant calculations # keep only unique nucleotide substitutions: # the same nucleotide substitution can be listed in multiple rows if it is within # more than one coding region, so remove columns with coding region information # then keep only unique rows cut -f 1-14 ERR6319327.variants.tsv | awk '!seen[$0]++ {print}' > unique_variants.tsv # filter variants that pass fisher's exact test grep "TRUE" unique_variants.tsv > passed_variants.tsv # calculate total number of variants variants_num=$(cat passed_variants.tsv | wc -l) if [ -z "$variants_num" ] ; then variants_num="0" ; fi echo $variants_num | tee VARIANT_NUM # calculate proportion of variants with allele frequencies between 0.6 and 0.9 # find number of variants at intermediate frequencies awk -F "\t" '{ if(($11 >= 0.6) && ($11 <= 0.9)) {print }}' passed_variants.tsv > intermediate_variants.tsv intermediates_num=$(cat intermediate_variants.tsv | wc -l) if [ -z "$intermediates_num" ] ; then intermediates_num="0" ; fi # if number of total variants is not zero, divide number of intermediate variants by total number of variants if [[ "$variants_num" -eq "0" ]]; then echo "Not computed: no variants detected" > PROPORTION_INTERMEDIATE else echo $intermediates_num $variants_num | awk '{ print $1/$2 }' > PROPORTION_INTERMEDIATE fi [2024-01-10 14:26:00,99] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.stats_n_coverage_primtrim:NA:1]: date | tee DATE samtools --version | head -n1 | tee VERSION samtools stats /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage_primtrim/inputs/1660897676/ERR6319327.primertrim.sorted.bam > ERR6319327.stats.txt samtools coverage /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage_primtrim/inputs/1660897676/ERR6319327.primertrim.sorted.bam -m -o ERR6319327.cov.hist samtools coverage /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage_primtrim/inputs/1660897676/ERR6319327.primertrim.sorted.bam -o ERR6319327.cov.txt samtools flagstat /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage_primtrim/inputs/1660897676/ERR6319327.primertrim.sorted.bam > ERR6319327.flagstat.txt coverage=$(cut -f 6 ERR6319327.cov.txt | tail -n 1) depth=$(cut -f 7 ERR6319327.cov.txt | tail -n 1) meanbaseq=$(cut -f 8 ERR6319327.cov.txt | tail -n 1) meanmapq=$(cut -f 9 ERR6319327.cov.txt | tail -n 1) if [ -z "$coverage" ] ; then coverage="0" ; fi if [ -z "$depth" ] ; then depth="0" ; fi if [ -z "$meanbaseq" ] ; then meanbaseq="0" ; fi if [ -z "$meanmapq" ] ; then meanmapq="0" ; fi echo $coverage | tee COVERAGE echo $depth | tee DEPTH echo $meanbaseq | tee MEANBASEQ echo $meanmapq | tee MEANMAPQ [2024-01-10 14:26:01,00] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.consensus:NA:1]: # date and version control date | tee DATE ivar version | head -n1 | tee IVAR_VERSION samtools --version | head -n1 | tee SAMTOOLS_VERSION # set reference genome if [[ ! -z "" ]]; then echo "User reference identified; will be utilized for alignement" ref_genome="" bwa index "" # move to primer_schemes dir; bwa fails if reference file not in this location else ref_genome="/artic-ncov2019/primer_schemes/nCoV-2019/V3/nCoV-2019.reference.fasta" fi # call consensus samtools mpileup \ --count-orphans \ -d 600000 \ --no-BAQ \ -Q 0 \ --reference ${ref_genome} \ /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-consensus/inputs/1660897676/ERR6319327.primertrim.sorted.bam | \ ivar consensus \ -p ERR6319327.consensus \ -q 20 \ -t 0.6 \ -m 100 \ -n N \ # clean up fasta header echo ">ERR6319327" > ERR6319327.ivar.consensus.fasta grep -v ">" ERR6319327.consensus.fa >> ERR6319327.ivar.consensus.fasta [2024-01-10 14:26:01,01] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.variant_call:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-variant_call/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-variant_call/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-variant_call:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-variant_call:delegated \ us-docker.pkg.dev/general-theiagen/staphb/ivar@sha256:3c3cb44b0db91912aa5b895c2bd9b7af71feebbae8dc959f0466c21b3d61c40b /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-variant_call/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-variant_call/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-variant_call/execution/docker_cid # return exit code exit $rc [2024-01-10 14:26:01,02] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.consensus:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-consensus/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-consensus/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-consensus:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-consensus:delegated \ us-docker.pkg.dev/general-theiagen/staphb/ivar@sha256:3c3cb44b0db91912aa5b895c2bd9b7af71feebbae8dc959f0466c21b3d61c40b /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-consensus/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-consensus/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-consensus/execution/docker_cid # return exit code exit $rc [2024-01-10 14:26:01,02] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.stats_n_coverage_primtrim:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage_primtrim/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage_primtrim/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage_primtrim:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage_primtrim:delegated \ us-docker.pkg.dev/general-theiagen/staphb/samtools@sha256:ef92cb5e490c75e66e89ab0a113865839e6954435d363af400eaa27177ff7c91 /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage_primtrim/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage_primtrim/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage_primtrim/execution/docker_cid # return exit code exit $rc [2024-01-10 14:26:05,77] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.variant_call:NA:1]: job id: 59413 [2024-01-10 14:26:05,77] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.stats_n_coverage_primtrim:NA:1]: job id: 59422 [2024-01-10 14:26:05,77] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.consensus:NA:1]: job id: 59418 [2024-01-10 14:26:05,77] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.stats_n_coverage_primtrim:NA:1]: Status change from - to Done [2024-01-10 14:26:05,78] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.variant_call:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:26:05,78] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.consensus:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:26:22,56] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.consensus:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:26:32,88] [info] BackgroundConfigAsyncJobExecutionActor [3700ee04ivar_consensus.variant_call:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:26:34,91] [info] 3700ee04-0c79-46e9-80b1-b086abe16c34-SubWorkflowActor-SubWorkflow-ivar_consensus:-1:1 [3700ee04]: Workflow ivar_consensus complete. Final Outputs: { "ivar_consensus.ivar_version_consensus": "iVar version 1.3.1", "ivar_consensus.read2_aligned": null, "ivar_consensus.assembly_method_nonflu": "BWA Version: 0.7.17-r1188; iVar version 1.3.1", "ivar_consensus.primer_bed_name": "artic-v3.primers.bed", "ivar_consensus.consensus_flagstat": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage/execution/ERR6319327.flagstat.txt", "ivar_consensus.ivar_variant_version": "iVar version 1.3.1", "ivar_consensus.samtools_version_primtrim": "samtools 1.12", "ivar_consensus.samtools_version_stats": "samtools 1.15", "ivar_consensus.primer_trimmed_read_percent": 13.48, "ivar_consensus.bwa_version": "BWA Version: 0.7.17-r1188", "ivar_consensus.consensus_stats": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-stats_n_coverage/execution/ERR6319327.stats.txt", "ivar_consensus.meanbaseq_trim": "38.8", "ivar_consensus.aligned_bam": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim/execution/ERR6319327.primertrim.sorted.bam", "ivar_consensus.samtools_version": "samtools 1.12", "ivar_consensus.meanmapq_trim": "60.0", "ivar_consensus.aligned_bai": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim/execution/ERR6319327.primertrim.sorted.bam.bai", "ivar_consensus.assembly_mean_coverage": "1092.55", "ivar_consensus.ivar_vcf": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-variant_call/execution/ERR6319327.variants.vcf", "ivar_consensus.ivar_version_primtrim": "iVar version 1.3.1", "ivar_consensus.ivar_tsv": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-variant_call/execution/ERR6319327.variants.tsv", "ivar_consensus.ivar_variant_proportion_intermediate": "0.047619", "ivar_consensus.consensus_n_variant_min_depth": 100, "ivar_consensus.read1_aligned": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-bwa/execution/ERR6319327_R1.fastq.gz", "ivar_consensus.samtools_version_consensus": "samtools 1.12", "ivar_consensus.assembly_fasta": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-consensus/execution/ERR6319327.ivar.consensus.fasta" } [2024-01-10 14:26:36,90] [info] WorkflowExecutionActor-4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1 [4cd7fc7e]: Starting theiacov_illumina_se.pangolin4, theiacov_illumina_se.nextclade, theiacov_illumina_se.sc2_gene_coverage, theiacov_illumina_se.consensus_qc [2024-01-10 14:26:40,95] [info] Assigned new job execution tokens to the following groups: 4cd7fc7e: 4 [2024-01-10 14:26:40,96] [warn] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.sc2_gene_coverage:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:26:40,97] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-primer_trim/execution/ERR6319327.primertrim.sorted.bam: Operation not permitted [2024-01-10 14:26:40,98] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.sc2_gene_coverage:NA:1]: samtools index /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam chr=$(samtools idxstats /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam | cut -f 1 | head -1) samtools coverage -r "${chr}:21563-25384" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam >> ERR6319327.cov.txt s_gene_depth=$(cut -f 7 ERR6319327.cov.txt | tail -n 1) # samtools outputs 3 columns; column 3 is the depth of coverage per nucleotide position, piped to awk to count the positions # above min_depth, then wc -l counts them all orf1ab=$(samtools depth -J -r "${chr}:266-21555" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) sgene=$(samtools depth -J -r "${chr}:21563-25384" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) orf3a=$(samtools depth -J -r "${chr}:25393-26220" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) egene=$(samtools depth -J -r "${chr}:26245-26472" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) mgene=$(samtools depth -J -r "${chr}:26523-27191" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) orf6=$(samtools depth -J -r "${chr}:27202-27387" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) orf7a=$(samtools depth -J -r "${chr}:27394-27759" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) orf7b=$(samtools depth -J -r "${chr}:27756-27887" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) orf8=$(samtools depth -J -r "${chr}:27894-28259" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) ngene=$(samtools depth -J -r "${chr}:28274-29533" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) orf10=$(samtools depth -J -r "${chr}:29558-29674" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/inputs/1660897676/ERR6319327.primertrim.sorted.bam | awk -F "\t" '{if ($3 > 100) print;}' | wc -l ) orf1ab_pc=$(python3 -c "print ( round( ($orf1ab / 21290 ) * 100, 2 ) )") sgene_pc=$(python3 -c "print ( round( ($sgene / 3822 ) * 100, 2 ) )") orf3a_pc=$(python3 -c "print ( round( ($orf3a / 828 ) * 100, 2 ) )") egene_pc=$(python3 -c "print ( round( ($egene / 228 ) * 100, 2 ) )") mgene_pc=$(python3 -c "print ( round( ($mgene / 669 ) * 100, 2 ) )") orf6_pc=$(python3 -c "print ( round( ($orf6 / 186 ) * 100, 2 ) )") orf7a_pc=$(python3 -c "print ( round( ($orf7a / 366 ) * 100, 2 ) )") orf7b_pc=$(python3 -c "print ( round( ($orf7b / 132 ) * 100, 2 ) )") orf8_pc=$(python3 -c "print ( round( ($orf8 / 366 ) * 100, 2 ) )") ngene_pc=$(python3 -c "print ( round( ($ngene / 1260 ) * 100, 2 ) )") orf10_pc=$(python3 -c "print ( round( ($orf10 / 117 ) * 100, 2 ) )") echo -e "#NOTE: THE VALUES BELOW ASSUME WUHAN-1 REFERENCE GENOME" > ERR6319327.percent_gene_coverage.tsv echo -e "Gene\tPercent_Coverage" >> ERR6319327.percent_gene_coverage.tsv echo -e "ORF1ab\t" $orf1ab_pc >> ERR6319327.percent_gene_coverage.tsv echo -e "S_gene\t" $sgene_pc >> ERR6319327.percent_gene_coverage.tsv echo -e "ORF3a\t" $orf3a_pc >> ERR6319327.percent_gene_coverage.tsv echo -e "E_gene\t" $egene_pc >> ERR6319327.percent_gene_coverage.tsv echo -e "M_gene\t" $mgene_pc >> ERR6319327.percent_gene_coverage.tsv echo -e "ORF6\t" $orf6_pc >> ERR6319327.percent_gene_coverage.tsv echo -e "ORF7a\t" $orf7a_pc >> ERR6319327.percent_gene_coverage.tsv echo -e "ORF7b\t" $orf7b_pc >> ERR6319327.percent_gene_coverage.tsv echo -e "ORF8\t" $orf8_pc >> ERR6319327.percent_gene_coverage.tsv echo -e "N_gene\t" $ngene_pc >> ERR6319327.percent_gene_coverage.tsv echo -e "ORF10\t" $orf10_pc >> ERR6319327.percent_gene_coverage.tsv if [ -z "s_gene_depth" ] ; then s_gene_depth="0"; fi echo $s_gene_depth | tee S_GENE_DEPTH echo $sgene_pc | tee S_GENE_PC [2024-01-10 14:26:41,00] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.sc2_gene_coverage:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage:delegated \ us-docker.pkg.dev/general-theiagen/staphb/samtools@sha256:ef92cb5e490c75e66e89ab0a113865839e6954435d363af400eaa27177ff7c91 /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-sc2_gene_coverage/execution/docker_cid # return exit code exit $rc [2024-01-10 14:26:41,17] [warn] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.consensus_qc:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:26:41,17] [warn] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.pangolin4:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:26:41,18] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-consensus_qc/inputs/1765922037/ERR6319327.ivar.consensus.fasta -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-consensus/execution/ERR6319327.ivar.consensus.fasta: Operation not permitted [2024-01-10 14:26:41,18] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-pangolin4/inputs/1765922037/ERR6319327.ivar.consensus.fasta -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-consensus/execution/ERR6319327.ivar.consensus.fasta: Operation not permitted [2024-01-10 14:26:41,18] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.consensus_qc:NA:1]: if [ -s "" ] ; then GENOME_LEN=$(grep -v ">" | tr --delete '\n' | wc -c) elif [ ] ; then GENOME_LEN= else # set SC2 default GENOME_LEN=29903 fi # capture date and version date | tee DATE num_N=$( grep -v ">" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-consensus_qc/inputs/1765922037/ERR6319327.ivar.consensus.fasta | grep -o 'N' | wc -l ) if [ -z "$num_N" ] ; then num_N="0" ; fi echo $num_N | tee NUM_N num_ACTG=$( grep -v ">" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-consensus_qc/inputs/1765922037/ERR6319327.ivar.consensus.fasta | grep -o -E "C|A|T|G" | wc -l ) if [ -z "$num_ACTG" ] ; then num_ACTG="0" ; fi echo $num_ACTG | tee NUM_ACTG # calculate percent coverage (Wu Han-1 genome length: 29903bp) python3 -c "print ( round( ($num_ACTG / $GENOME_LEN ) * 100, 2 ) )" | tee PERCENT_REF_COVERAGE num_degenerate=$( grep -v ">" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-consensus_qc/inputs/1765922037/ERR6319327.ivar.consensus.fasta | grep -o -E "B|D|E|F|H|I|J|K|L|M|O|P|Q|R|S|U|V|W|X|Y|Z" | wc -l ) if [ -z "$num_degenerate" ] ; then num_degenerate="0" ; fi echo $num_degenerate | tee NUM_DEGENERATE num_total=$( grep -v ">" /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-consensus_qc/inputs/1765922037/ERR6319327.ivar.consensus.fasta | grep -o -E '[A-Z]' | wc -l ) if [ -z "$num_total" ] ; then num_total="0" ; fi echo $num_total | tee NUM_TOTAL [2024-01-10 14:26:41,18] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.pangolin4:NA:1]: set -e # date and version capture date | tee DATE { pangolin --all-versions && usher --version; } | tr '\n' ';' | cut -f -6 -d ';' | tee VERSION_PANGOLIN_ALL pangolin "/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-pangolin4/inputs/1765922037/ERR6319327.ivar.consensus.fasta" \ \ --min-length 10000 \ --max-ambig 0.5 \ --expanded-lineage \ \ \ --outfile "ERR6319327.pangolin_report.csv" \ --verbose \ python3 < /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-consensus/execution/ERR6319327.ivar.consensus.fasta: Operation not permitted [2024-01-10 14:26:41,29] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.nextclade:NA:1]: NEXTCLADE_VERSION="$(nextclade --version)" echo $NEXTCLADE_VERSION > NEXTCLADE_VERSION nextclade dataset get --name="sars-cov-2" --reference="MN908947" --tag="2023-09-21T12:00:00Z" -o nextclade_dataset_dir --verbose set -e nextclade run \ --input-dataset=nextclade_dataset_dir/ \ \ \ \ \ \ \ --output-json "ERR6319327.ivar.consensus".nextclade.json \ --output-tsv "ERR6319327.ivar.consensus".nextclade.tsv \ --output-tree "ERR6319327.ivar.consensus".nextclade.auspice.json \ --output-all=. \ "/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-nextclade/inputs/1765922037/ERR6319327.ivar.consensus.fasta" [2024-01-10 14:26:41,30] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.nextclade:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-nextclade/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-nextclade/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-nextclade:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-nextclade:delegated \ us-docker.pkg.dev/general-theiagen/nextstrain/nextclade@sha256:f4440021c7f854433bd56433024f38199de27e0ef657e11d5fb28aad4f265a48 /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-nextclade/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-nextclade/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-nextclade/execution/docker_cid # return exit code exit $rc [2024-01-10 14:26:45,76] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.pangolin4:NA:1]: job id: 60138 [2024-01-10 14:26:45,76] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.nextclade:NA:1]: job id: 60191 [2024-01-10 14:26:45,77] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.nextclade:NA:1]: Status change from - to Done [2024-01-10 14:26:45,77] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.consensus_qc:NA:1]: job id: 60141 [2024-01-10 14:26:45,77] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.sc2_gene_coverage:NA:1]: job id: 60066 [2024-01-10 14:26:45,77] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.sc2_gene_coverage:NA:1]: Status change from - to Done [2024-01-10 14:26:45,77] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.pangolin4:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:26:45,77] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.consensus_qc:NA:1]: Status change from - to Done [2024-01-10 14:26:47,10] [info] WorkflowExecutionActor-4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1 [4cd7fc7e]: Starting theiacov_illumina_se.vadr, theiacov_illumina_se.nextclade_output_parser [2024-01-10 14:26:50,95] [info] Assigned new job execution tokens to the following groups: 4cd7fc7e: 2 [2024-01-10 14:26:51,11] [warn] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.vadr:NA:1]: Unrecognized runtime attribute keys: dx_instance_type, cpu, memory [2024-01-10 14:26:51,11] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-vadr/inputs/1765922037/ERR6319327.ivar.consensus.fasta -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-ivar_consensus/ivar_consensus/3700ee04-0c79-46e9-80b1-b086abe16c34/call-consensus/execution/ERR6319327.ivar.consensus.fasta: Operation not permitted [2024-01-10 14:26:51,12] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.vadr:NA:1]: set -e if [ 29377 -gt 10000 ]; then # remove terminal ambiguous nucleotides /opt/vadr/vadr/miniscripts/fasta-trim-terminal-ambigs.pl \ "/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-vadr/inputs/1765922037/ERR6319327.ivar.consensus.fasta" \ --minlen 50 \ --maxlen 30000 \ > "ERR6319327.ivar.consensus_trimmed.fasta" # run VADR # --split and --cpu must be used in conjuction v-annotate.pl \ --split --cpu 2 \ --noseqnamemax --glsearch -s -r --nomisc --mkey sarscov2 --lowsim5seq 6 --lowsim3seq 6 --alt_fail lowscore,insertnn,deletinn --out_allfasta \ "ERR6319327.ivar.consensus_trimmed.fasta" \ "ERR6319327.ivar.consensus" # package everything for output tar -C "ERR6319327.ivar.consensus" -czvf "ERR6319327.ivar.consensus.vadr.tar.gz" . # package up FASTA files into zip file for output. Note: this will work whether the --out_allfasta flag is included or not (there are just more when the option is used) mkdir -v vadr_fasta_files cp -v ERR6319327.ivar.consensus/*.fa vadr_fasta_files zip ERR6319327.ivar.consensus_vadr-fasta-files.zip vadr_fasta_files/*.fa # prep alerts into a tsv file for parsing cut -f 5 "ERR6319327.ivar.consensus/ERR6319327.ivar.consensus.vadr.alt.list" | tail -n +2 > "ERR6319327.ivar.consensus.vadr.alerts.tsv" cat "ERR6319327.ivar.consensus.vadr.alerts.tsv" | wc -l > NUM_ALERTS else echo "VADR skipped due to poor assembly; assembly length (unambiguous) = 29377" > NUM_ALERTS fi [2024-01-10 14:26:51,13] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.vadr:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-vadr/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-vadr/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-vadr:/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-vadr:delegated \ us-docker.pkg.dev/general-theiagen/staphb/vadr@sha256:f15a73a6f6e0802c3a9c05e7c602740e7029a2e1ccc14aa6935b459be87c98d7 /cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-vadr/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-vadr/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-vadr/execution/docker_cid # return exit code exit $rc [2024-01-10 14:26:51,35] [warn] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.nextclade_output_parser:NA:1]: Unrecognized runtime attribute keys: disk, dx_instance_type, disks, cpu, memory [2024-01-10 14:26:51,35] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-nextclade_output_parser/inputs/-990009139/ERR6319327.ivar.consensus.nextclade.tsv -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-nextclade/execution/ERR6319327.ivar.consensus.nextclade.tsv: Operation not permitted [2024-01-10 14:26:51,36] [info] BackgroundConfigAsyncJobExecutionActor [4cd7fc7etheiacov_illumina_se.nextclade_output_parser:NA:1]: # Set WDL input variable to input.tsv file cat "/cromwell-executions/theiacov_illumina_se/4cd7fc7e-48f0-4bc3-98ba-ca96aee775e1/call-nextclade_output_parser/inputs/-990009139/ERR6319327.ivar.consensus.nextclade.tsv" > input.tsv touch TAMIFLU_AASUBS # Parse outputs using python3 python3 <

✅ theiacov_illumina_se run via cromwell was successful & exit code was 0

One more to go - theiacov_ont

kapsakcj commented 9 months ago

theiacov_ont via cromwell

command used:

$ cromwell run -i ./tests/inputs/theiacov/wf_theiacov_ont.json -m metadata.json ./workflows/theiacov/wf_theiacov_ont.wdl
most of the output of cromwell ```bash INFO: Columns REPORTABLE_RESULT(TINYINT) added to SYMBOL Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/top_level_output.xml::top_level_output::chrisl ran successfully in 4ms Running Changeset: changesets/workflow_execution_aux_not_null.xml::workflow_execution_aux_not_null::tjeandet Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_EXECUTION_AUX.WORKFLOW_EXECUTION_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_EXECUTION_AUX.WDL_SOURCE Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_EXECUTION_AUX.JSON_INPUTS Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_EXECUTION_AUX.WORKFLOW_OPTIONS Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/workflow_execution_aux_not_null.xml::workflow_execution_aux_not_null::tjeandet ran successfully in 10ms Running Changeset: changesets/call_result_caching.xml::call_result_caching::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Columns ALLOWS_RESULT_REUSE(BOOLEAN),DOCKER_IMAGE_HASH(VARCHAR(100)),RESULTS_CLONED_FROM(INT),EXECUTION_HASH(VARCHAR(100)) added to EXECUTION Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index HASH_INDEX created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Columns HASH(VARCHAR(100)) added to SYMBOL Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION (RESULTS_CLONED_FROM) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/call_result_caching.xml::call_result_caching::chrisl ran successfully in 15ms Running Changeset: changesets/events_table.xml::execution_event_table::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table EXECUTION_EVENT created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION_EVENT (EXECUTION_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to EXECUTION_EVENT(EXECUTION_ID, DESCRIPTION) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/events_table.xml::execution_event_table::chrisl ran successfully in 9ms Running Changeset: changesets/sync_not_null_constraints.xml::job_not_null_columns::kshakir Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Null constraint dropped from JES_JOB.JES_RUN_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Null constraint dropped from JES_JOB.JES_STATUS Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Null constraint dropped from SGE_JOB.SGE_JOB_NUMBER Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/sync_not_null_constraints.xml::job_not_null_columns::kshakir ran successfully in 5ms Running Changeset: changesets/sync_not_null_constraints.xml::events_end_dt_not_null::kshakir Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: NOT NULL constraint has been added to EXECUTION_EVENT.END_DT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/sync_not_null_constraints.xml::events_end_dt_not_null::kshakir ran successfully in 2ms Running Changeset: changesets/sge_job_execution_unique_key.xml::sge_job_execution_unique_key::kshakir Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to SGE_JOB(EXECUTION_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/sge_job_execution_unique_key.xml::sge_job_execution_unique_key::kshakir ran successfully in 3ms Running Changeset: changesets/add_attempt_in_execution.xml::add-attempt-in-execution::tjeandet Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Columns ATTEMPT(INT) added to EXECUTION Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: NOT NULL constraint has been added to EXECUTION.ATTEMPT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key FK_EXECUTION_WORKFLOW dropped Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint UK_WORKFLOW_CALL_INDEX dropped from EXECUTION Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to EXECUTION(WORKFLOW_EXECUTION_ID, CALL_FQN, IDX, ATTEMPT) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION (WORKFLOW_EXECUTION_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/add_attempt_in_execution.xml::add-attempt-in-execution::tjeandet ran successfully in 14ms Running Changeset: changesets/execution_backend_info.xml::CREATE_EXECUTION_INFO::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table EXECUTION_INFO created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::CREATE_EXECUTION_INFO::mcovarr ran successfully in 3ms Running Changeset: changesets/execution_backend_info.xml::add_foreign_key_constraints::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to EXECUTION_INFO (EXECUTION_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::add_foreign_key_constraints::mcovarr ran successfully in 2ms Running Changeset: changesets/execution_backend_info.xml::add_backend_type_column::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Columns BACKEND_TYPE(VARCHAR(255)) added to EXECUTION Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::add_backend_type_column::mcovarr ran successfully in 3ms Running Changeset: changesets/execution_backend_info.xml::insert_data_from_backend_tables::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::insert_data_from_backend_tables::mcovarr ran successfully in 15ms Running Changeset: changesets/execution_backend_info.xml::set_backend_type::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::set_backend_type::mcovarr ran successfully in 7ms Running Changeset: changesets/execution_backend_info.xml::drop_job_tables::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table JES_JOB dropped Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table SGE_JOB dropped Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table LOCAL_JOB dropped Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::drop_job_tables::mcovarr ran successfully in 8ms Running Changeset: changesets/execution_backend_info.xml::execution_info_uniqueness_constraint::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to EXECUTION_INFO(EXECUTION_ID, INFO_KEY) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::execution_info_uniqueness_constraint::mcovarr ran successfully in 4ms Running Changeset: changesets/execution_backend_info.xml::execution_backend_type_not_null::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: NOT NULL constraint has been added to EXECUTION.BACKEND_TYPE Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/execution_backend_info.xml::execution_backend_type_not_null::mcovarr ran successfully in 3ms Running Changeset: changesets/runtime_attributes_table.xml::runtime-attributes-table::tjeandet Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table RUNTIME_ATTRIBUTES created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to RUNTIME_ATTRIBUTES (EXECUTION_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to RUNTIME_ATTRIBUTES(EXECUTION_ID, ATTRIBUTE_NAME) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/runtime_attributes_table.xml::runtime-attributes-table::tjeandet ran successfully in 8ms Running Changeset: changesets/failure_table.xml::failure_table::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table FAILURE_EVENT created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to FAILURE_EVENT (EXECUTION_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to FAILURE_EVENT (WORKFLOW_EXECUTION_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/failure_table.xml::failure_table::chrisl ran successfully in 9ms Running Changeset: changesets/metadata_journal.xml::metadata_journal::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table METADATA_JOURNAL created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal.xml::metadata_journal::mcovarr ran successfully in 3ms Running Changeset: changesets/metadata_journal.xml::metadata_workflow_index::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index METADATA_WORKFLOW_IDX created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal.xml::metadata_workflow_index::mcovarr ran successfully in 2ms Running Changeset: changesets/metadata_journal.xml::metadata_job_index::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index METADATA_JOB_IDX created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal.xml::metadata_job_index::mcovarr ran successfully in 3ms Running Changeset: changesets/metadata_journal.xml::metadata_job_and_key_index::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index METADATA_JOB_AND_KEY_IDX created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal.xml::metadata_job_and_key_index::mcovarr ran successfully in 2ms Running Changeset: changesets/metadata_typed_values.xml::metadata_typed_values::tjeandet Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Columns METADATA_VALUE_TYPE(VARCHAR(10)) added to METADATA_JOURNAL Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/metadata_typed_values.xml::metadata_typed_values::tjeandet ran successfully in 3ms Running Changeset: changesets/workflow_metadata_summary.xml::metadata_journal_id_int_to_big_int::kshakir Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: METADATA_JOURNAL.METADATA_JOURNAL_ID datatype was changed to BIGINT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Auto-increment added to METADATA_JOURNAL.METADATA_JOURNAL_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::metadata_journal_id_int_to_big_int::kshakir ran successfully in 4ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_summary::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_summary::mcovarr ran successfully in 4ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_uuid_idx::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_METADATA_SUMMARY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_uuid_idx::mcovarr ran successfully in 3ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_name_idx::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_NAME_IDX created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_name_idx::mcovarr ran successfully in 3ms Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_status_idx::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_STATUS_IDX created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/workflow_metadata_summary.xml::workflow_metadata_status_idx::mcovarr ran successfully in 2ms Running Changeset: changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_subsecond_timestamp::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: METADATA_JOURNAL.METADATA_TIMESTAMP datatype was changed to DATETIME(6) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_subsecond_timestamp::mcovarr ran successfully in 2ms Running Changeset: changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_timestamp_not_null::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: NOT NULL constraint has been added to METADATA_JOURNAL.METADATA_TIMESTAMP Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_timestamp_not_null::mcovarr ran successfully in 2ms Running Changeset: changesets/workflow_store.xml::WORKFLOW_STORE::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table WORKFLOW_STORE created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store.xml::WORKFLOW_STORE::chrisl ran successfully in 5ms Running Changeset: changesets/workflow_store.xml::workflow_store_uuid_index::kshakir Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_STORE(WORKFLOW_UUID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store.xml::workflow_store_uuid_index::kshakir ran successfully in 3ms Running Changeset: changesets/workflow_store.xml::workflow_store_state_index::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index WORKFLOW_STORE_STATE_IDX created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store.xml::workflow_store_state_index::chrisl ran successfully in 2ms Running Changeset: changesets/backend_KV_Store.xml::BACKEND_KV_STORE::rmunshi Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table BACKEND_KV_STORE created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/backend_KV_Store.xml::BACKEND_KV_STORE::rmunshi ran successfully in 4ms Running Changeset: changesets/backend_KV_Store.xml::backend_KV_store_job_key_constraint::rmunshi Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to BACKEND_KV_STORE(WORKFLOW_EXECUTION_UUID, CALL_FQN, JOB_SCATTER_INDEX, JOB_RETRY_ATTEMPT, STORE_KEY) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/backend_KV_Store.xml::backend_KV_store_job_key_constraint::rmunshi ran successfully in 3ms Running Changeset: changesets/job_store.xml::JOB_STORE::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table JOB_STORE created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/job_store.xml::JOB_STORE::chrisl ran successfully in 4ms Running Changeset: changesets/job_store.xml::job_store_uuid_index::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index JOB_STORE_UUID_IDX created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/job_store.xml::job_store_uuid_index::chrisl ran successfully in 2ms Running Changeset: changesets/job_store.xml::job_store_jobkey_index::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE(WORKFLOW_UUID, CALL_FQN, JOB_SCATTER_INDEX, JOB_RETRY_ATTEMPT) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/job_store.xml::job_store_jobkey_index::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_metainfo::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_METAINFO created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_metainfo::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_metainfo_uniqueness::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_RESULT_METAINFO(WORKFLOW_UUID, CALL_FQN, JOB_SCATTER_INDEX) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_metainfo_uniqueness::chrisl ran successfully in 2ms Running Changeset: changesets/callcaching.xml::call_caching_hashes::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table CALL_CACHING_HASH created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_hashes::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_hashes_uniqueness::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_HASH(HASH_KEY, RESULT_METAINFO_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_hashes_uniqueness::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_hashes_uniqueness_foreign_key::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH (RESULT_METAINFO_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_hashes_uniqueness_foreign_key::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_SIMPLETON created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_simpletons::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons_uniqueness::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_RESULT_SIMPLETON(SIMPLETON_KEY, RESULT_METAINFO_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_simpletons_uniqueness::chrisl ran successfully in 3ms Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons_foreign_key::chrisl Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_RESULT_SIMPLETON (RESULT_METAINFO_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/callcaching.xml::call_caching_result_simpletons_foreign_key::chrisl ran successfully in 3ms Running Changeset: changesets/call_caching_allow_result_reuse_fix.xml::call_caching_fix_boolean::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: CALL_CACHING_RESULT_METAINFO.ALLOW_RESULT_REUSE datatype was changed to BOOLEAN Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_allow_result_reuse_fix.xml::call_caching_fix_boolean::mcovarr ran successfully in 2ms Running Changeset: changesets/call_caching_allow_result_reuse_fix.xml::call_caching_reuse_default::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Default value added to CALL_CACHING_RESULT_METAINFO.ALLOW_RESULT_REUSE Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_allow_result_reuse_fix.xml::call_caching_reuse_default::mcovarr ran successfully in 3ms Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table JOB_STORE_RESULT_SIMPLETON created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_result_simpletons::mcovarr ran successfully in 5ms Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons_uniqueness::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE_RESULT_SIMPLETON(SIMPLETON_KEY, JOB_STORE_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_result_simpletons_uniqueness::mcovarr ran successfully in 4ms Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons_foreign_key::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_RESULT_SIMPLETON (JOB_STORE_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_result_simpletons_foreign_key::mcovarr ran successfully in 3ms Running Changeset: changesets/job_store_simpletons.xml::job_store_remove_job_output::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column JOB_STORE.JOB_OUTPUT dropped Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/job_store_simpletons.xml::job_store_remove_job_output::mcovarr ran successfully in 4ms Running Changeset: changesets/restart_and_recover_migration.xml::WORKFLOW_STORE_MIGRATION::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::WORKFLOW_STORE_MIGRATION::mcovarr ran successfully in 6ms Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_MIGRATION::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::JOB_STORE_MIGRATION::mcovarr ran successfully in 11ms Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_VALUE_NULLABLE::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Null constraint dropped from JOB_STORE_RESULT_SIMPLETON.SIMPLETON_VALUE Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_VALUE_NULLABLE::mcovarr ran successfully in 25ms Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_MIGRATION::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: WORKFLOW_EXECUTION + EXECUTION + SYMBOL + JOB_STORE -> JOB_STORE_RESULT_SIMPLETON migration complete. Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_MIGRATION::mcovarr ran successfully in 3ms Running Changeset: changesets/restart_and_recover_migration.xml::BACKEND_KV_STORE_MIGRATION::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::BACKEND_KV_STORE_MIGRATION::mcovarr ran successfully in 4ms Running Changeset: changesets/restart_and_recover_migration.xml::WORKFLOW_OPTIONS_RENAME_MIGRATION::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Workflow option renaming migration complete. Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/restart_and_recover_migration.xml::WORKFLOW_OPTIONS_RENAME_MIGRATION::mcovarr ran successfully in 3ms Running Changeset: changesets/summary_status_table.xml::summary_status_table::kshakir Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table SUMMARY_STATUS created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/summary_status_table.xml::summary_status_table::kshakir ran successfully in 3ms Running Changeset: changesets/summary_status_table.xml::summary_status_summarized_table_name_index::kshakir Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS(SUMMARY_TABLE_NAME, SUMMARIZED_TABLE_NAME) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/summary_status_table.xml::summary_status_summarized_table_name_index::kshakir ran successfully in 2ms Running Changeset: changesets/standardize_column_names.xml::workflow_execution_uuid::rmunshi Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column CALL_CACHING_RESULT_METAINFO.WORKFLOW_UUID renamed to WORKFLOW_EXECUTION_UUID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column WORKFLOW_STORE.WORKFLOW_UUID renamed to WORKFLOW_EXECUTION_UUID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column JOB_STORE.WORKFLOW_UUID renamed to WORKFLOW_EXECUTION_UUID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names.xml::workflow_execution_uuid::rmunshi ran successfully in 4ms Running Changeset: changesets/standardize_column_names.xml::call_job_identifiers::rmunshi Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column METADATA_JOURNAL.METADATA_CALL_FQN renamed to CALL_FQN Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column METADATA_JOURNAL.METADATA_CALL_INDEX renamed to JOB_SCATTER_INDEX Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column METADATA_JOURNAL.METADATA_CALL_ATTEMPT renamed to JOB_RETRY_ATTEMPT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names.xml::call_job_identifiers::rmunshi ran successfully in 4ms Running Changeset: changesets/embiggen_metadata_value.xml::entry_or_journal_existence_xor::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/embiggen_metadata_value.xml::entry_or_journal_existence_xor::mcovarr ran successfully in 16ms Running Changeset: changesets/embiggen_metadata_value.xml::embiggen_metadata_entry::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Marking ChangeSet: changesets/embiggen_metadata_value.xml::embiggen_metadata_entry::mcovarr ran despite precondition failure due to onFail='MARK_RAN': changelog.xml : Table PUBLIC.METADATA_ENTRY does not exist Running Changeset: changesets/embiggen_metadata_value.xml::embiggen_metadata_journal::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: METADATA_JOURNAL.METADATA_VALUE datatype was changed to LONGTEXT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/embiggen_metadata_value.xml::embiggen_metadata_journal::mcovarr ran successfully in 5ms Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus::rmunshi Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table CALL_CACHING_JOB_DETRITUS created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_job_detritus.xml::call_caching_job_detritus::rmunshi ran successfully in 3ms Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus_key_uniqueness::rmunshi Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_JOB_DETRITUS(JOB_DETRITUS_KEY, RESULT_METAINFO_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_job_detritus.xml::call_caching_job_detritus_key_uniqueness::rmunshi ran successfully in 3ms Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus_foreign_key::rmunshi Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_JOB_DETRITUS (RESULT_METAINFO_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_job_detritus.xml::call_caching_job_detritus_foreign_key::rmunshi ran successfully in 3ms Running Changeset: changesets/job_store_tinyints.xml::job_store_fix_job_successful::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: JOB_STORE.JOB_SUCCESSFUL datatype was changed to BOOLEAN Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/job_store_tinyints.xml::job_store_fix_job_successful::mcovarr ran successfully in 2ms Running Changeset: changesets/job_store_tinyints.xml::job_store_fix_job_retryable_failure::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: JOB_STORE.RETRYABLE_FAILURE datatype was changed to BOOLEAN Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/job_store_tinyints.xml::job_store_fix_job_retryable_failure::mcovarr ran successfully in 2ms Running Changeset: changesets/job_store_tinyints.xml::job_store_not_nullable_job_successful::mcovarr Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: NOT NULL constraint has been added to JOB_STORE.JOB_SUCCESSFUL Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/job_store_tinyints.xml::job_store_not_nullable_job_successful::mcovarr ran successfully in 2ms Running Changeset: changesets/standardize_column_names_again.xml::standardize_column_names_again::kshakir Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index JOB_STORE_UUID_IDX dropped from table JOB_STORE Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_NAME_IDX dropped from table WORKFLOW_METADATA_SUMMARY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index WORKFLOW_METADATA_STATUS_IDX dropped from table WORKFLOW_METADATA_SUMMARY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index WORKFLOW_STORE_STATE_IDX dropped from table WORKFLOW_STORE Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_JOB_DETRITUS_KEY dropped from CALL_CACHING_JOB_DETRITUS Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_RESULT_METAINFO dropped from CALL_CACHING_RESULT_METAINFO Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_HASH dropped from CALL_CACHING_HASH Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint UK_CALL_CACHING_RESULT_SIMPLETON dropped from CALL_CACHING_RESULT_SIMPLETON Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint JOB_STORE_JOBKEY_IDX dropped from JOB_STORE Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint UK_BACKEND_KV_STORE_KEY dropped from BACKEND_KV_STORE Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint UK_JOB_STORE_RESULT_SIMPLETON dropped from JOB_STORE_RESULT_SIMPLETON Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint SUMMARY_STATUS_SUMMARY_TABLE_NAME_SUMMARIZED_TABLE_NAME_INDEX dropped from SUMMARY_STATUS Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint WORKFLOW_METADATA_UUID_IDX dropped from WORKFLOW_METADATA_SUMMARY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint WORKFLOW_STORE_UUID_IDX dropped from WORKFLOW_STORE Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key CCJD_RESULT_METAINFO_ID_FK dropped Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key CCH_RESULT_METAINFO_ID_FK dropped Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key CCRS_RESULT_METAINFO_ID_FK dropped Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key JSRS_JOB_STORE_FK dropped Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table CALL_CACHING_JOB_DETRITUS renamed to CALL_CACHING_DETRITUS_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_METAINFO renamed to CALL_CACHING_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table CALL_CACHING_HASH renamed to CALL_CACHING_HASH_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table CALL_CACHING_RESULT_SIMPLETON renamed to CALL_CACHING_SIMPLETON_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table BACKEND_KV_STORE renamed to JOB_KEY_VALUE_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table JOB_STORE renamed to JOB_STORE_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table JOB_STORE_RESULT_SIMPLETON renamed to JOB_STORE_SIMPLETON_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table METADATA_JOURNAL renamed to METADATA_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table SUMMARY_STATUS renamed to SUMMARY_STATUS_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY renamed to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table WORKFLOW_STORE renamed to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_JOB_DETRITUS_ID renamed to CALL_CACHING_DETRITUS_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.JOB_DETRITUS_KEY renamed to DETRITUS_KEY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.JOB_DETRITUS_VALUE renamed to DETRITUS_VALUE Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column CALL_CACHING_DETRITUS_ENTRY.RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column CALL_CACHING_ENTRY.CALL_CACHING_RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column CALL_CACHING_ENTRY.JOB_SCATTER_INDEX renamed to JOB_INDEX Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column CALL_CACHING_ENTRY.CALL_FQN renamed to CALL_FULLY_QUALIFIED_NAME Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column CALL_CACHING_HASH_ENTRY.RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ID renamed to CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column CALL_CACHING_SIMPLETON_ENTRY.RESULT_METAINFO_ID renamed to CALL_CACHING_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_RESULT_SIMPLETON_ID renamed to CALL_CACHING_SIMPLETON_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.JOB_RETRY_ATTEMPT renamed to JOB_ATTEMPT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.JOB_SCATTER_INDEX renamed to JOB_INDEX Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.BACKEND_KV_STORE_ID renamed to JOB_KEY_VALUE_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column JOB_KEY_VALUE_ENTRY.CALL_FQN renamed to CALL_FULLY_QUALIFIED_NAME Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.JOB_RETRY_ATTEMPT renamed to JOB_ATTEMPT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.JOB_SCATTER_INDEX renamed to JOB_INDEX Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.JOB_STORE_ID renamed to JOB_STORE_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column JOB_STORE_ENTRY.CALL_FQN renamed to CALL_FULLY_QUALIFIED_NAME Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_ID renamed to JOB_STORE_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_RESULT_SIMPLETON_ID renamed to JOB_STORE_SIMPLETON_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.SUMMARY_STATUS_ID renamed to SUMMARY_STATUS_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column WORKFLOW_METADATA_SUMMARY_ENTRY.END_DT renamed to END_TIMESTAMP Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column WORKFLOW_METADATA_SUMMARY_ENTRY.START_DT renamed to START_TIMESTAMP Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column WORKFLOW_METADATA_SUMMARY_ENTRY.WORKFLOW_METADATA_SUMMARY_ID renamed to WORKFLOW_METADATA_SUMMARY_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Column WORKFLOW_STORE_ENTRY.WORKFLOW_STORE_ID renamed to WORKFLOW_STORE_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_DETRITUS_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_HASH_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key dropped from CALL_CACHING_SIMPLETON_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key dropped from JOB_KEY_VALUE_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key dropped from JOB_STORE_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key dropped from JOB_STORE_SIMPLETON_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key dropped from SUMMARY_STATUS_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key dropped from WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key dropped from WORKFLOW_STORE_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_DETRITUS_ENTRY (CALL_CACHING_DETRITUS_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_HASH_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key added to CALL_CACHING_SIMPLETON_ENTRY (CALL_CACHING_SIMPLETON_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key added to JOB_KEY_VALUE_ENTRY (JOB_KEY_VALUE_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key added to JOB_STORE_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_SIMPLETON_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key added to SUMMARY_STATUS_ENTRY (SUMMARY_STATUS_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key added to WORKFLOW_METADATA_SUMMARY_ENTRY (WORKFLOW_METADATA_SUMMARY_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Primary key added to WORKFLOW_STORE_ENTRY (WORKFLOW_STORE_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_DETRITUS_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_ENTRY.CALL_CACHING_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_SIMPLETON_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Auto-increment added to JOB_KEY_VALUE_ENTRY.JOB_KEY_VALUE_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_ENTRY.JOB_STORE_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_SIMPLETON_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Auto-increment added to SUMMARY_STATUS_ENTRY.SUMMARY_STATUS_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Auto-increment added to WORKFLOW_METADATA_SUMMARY_ENTRY.WORKFLOW_METADATA_SUMMARY_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Auto-increment added to WORKFLOW_STORE_ENTRY.WORKFLOW_STORE_ENTRY_ID Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_DETRITUS_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_SIMPLETON_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_DETRITUS_ENTRY(CALL_CACHING_ENTRY_ID, DETRITUS_KEY) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_HASH_ENTRY(CALL_CACHING_ENTRY_ID, HASH_KEY) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_SIMPLETON_ENTRY(CALL_CACHING_ENTRY_ID, SIMPLETON_KEY) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to JOB_KEY_VALUE_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX, JOB_ATTEMPT, STORE_KEY) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX, JOB_ATTEMPT) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to JOB_STORE_SIMPLETON_ENTRY(JOB_STORE_ENTRY_ID, SIMPLETON_KEY) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS_ENTRY(SUMMARY_TABLE_NAME, SUMMARIZED_TABLE_NAME) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_METADATA_SUMMARY_ENTRY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_STORE_ENTRY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index IX_JOB_STORE_ENTRY_WEU created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WN created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WS created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index IX_WORKFLOW_STORE_ENTRY_WS created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_again.xml::standardize_column_names_again::kshakir ran successfully in 185ms Running Changeset: changesets/rename_workflow_options_in_metadata.xml::rename_workflow_options_in_metadata::tjeandet [2024-01-10 14:30:56,87] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000 [2024-01-10 14:30:56,89] [info] [RenameWorkflowOptionsInMetadata] 100% Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: RenameWorkflowOptionsInMetadata complete. Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/rename_workflow_options_in_metadata.xml::rename_workflow_options_in_metadata::tjeandet ran successfully in 28ms Running Changeset: changesets/encrypt_and_clear_workflow_options.xml::encrypt_workflow_store_entry_workflow_options::kshakir Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: EncryptWorkflowStoreEntryWorkflowOptions complete. Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/encrypt_and_clear_workflow_options.xml::encrypt_workflow_store_entry_workflow_options::kshakir ran successfully in 2ms Running Changeset: changesets/encrypt_and_clear_workflow_options.xml::clear_metadata_entry_workflow_options::kshakir Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ClearMetadataEntryWorkflowOptions complete. Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/encrypt_and_clear_workflow_options.xml::clear_metadata_entry_workflow_options::kshakir ran successfully in 1ms Running Changeset: changesets/sub_workflow_store.xml::SUB_WORKFLOW_STORE_ENTRY::tjeandet Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Table SUB_WORKFLOW_STORE_ENTRY created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::SUB_WORKFLOW_STORE_ENTRY::tjeandet ran successfully in 4ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_uuid_index::tjeandet Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Index IX_SUB_WORKFLOW_STORE_ENTRY_PWEU created Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_uuid_index::tjeandet ran successfully in 3ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_jobkey_index::tjeandet Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to SUB_WORKFLOW_STORE_ENTRY(PARENT_WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, CALL_INDEX, CALL_ATTEMPT) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_jobkey_index::tjeandet ran successfully in 3ms Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_root_workflow_fk::tjeandet Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to SUB_WORKFLOW_STORE_ENTRY (ROOT_WORKFLOW_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/sub_workflow_store.xml::sub_workflow_store_root_workflow_fk::tjeandet ran successfully in 3ms Running Changeset: changesets/workflow_store_imports_file.xml::workflow-store-imports-file::cjllanwarne Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Columns IMPORTS_ZIP(BLOB) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_imports_file.xml::workflow-store-imports-file::cjllanwarne ran successfully in 4ms Running Changeset: changesets/workflow_store_labels_file.xml::workflow-store-labels-file::cjllanwarne Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Columns CUSTOM_LABELS(LONGVARCHAR) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_labels_file.xml::workflow-store-labels-file::cjllanwarne ran successfully in 3ms Running Changeset: changesets/embiggen_detritus_value.xml::embiggen_call_caching_detritus_entry::rmunshi Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE datatype was changed to LONGVARCHAR Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/embiggen_detritus_value.xml::embiggen_call_caching_detritus_entry::rmunshi ran successfully in 3ms Running Changeset: changesets/standardize_column_names_patches.xml::standardize_column_names_patches::kshakir Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_HASH_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key FK_JOB_STORE_RESULT_SIMPLETON_JOB_STORE_ENTRY_ID dropped Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key FK_SUB_WORKFLOW_STORE_ROOT_WORKFLOW_ID_WORKFLOW_STORE_ENTRY_ID dropped Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint UC_CALL_CACHING_ENTRY_WEU_CQFN_JI dropped from CALL_CACHING_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint UC_CALL_CACHING_HASH_ENTRY_CCEI dropped from CALL_CACHING_HASH_ENTRY Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_ENTRY(WORKFLOW_EXECUTION_UUID, CALL_FULLY_QUALIFIED_NAME, JOB_INDEX) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Unique constraint added to CALL_CACHING_HASH_ENTRY(CALL_CACHING_ENTRY_ID, HASH_KEY) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: Foreign key constraint added to SUB_WORKFLOW_STORE_ENTRY (ROOT_WORKFLOW_ID) Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_patches.xml::standardize_column_names_patches::kshakir ran successfully in 20ms Running Changeset: changesets/standardize_column_names_patches.xml::hsqldb_longvarchar_to_longtext::kshakir Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE datatype was changed to LONGTEXT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.SIMPLETON_VALUE datatype was changed to LONGTEXT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: JOB_STORE_ENTRY.EXCEPTION_MESSAGE datatype was changed to LONGTEXT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: JOB_STORE_SIMPLETON_ENTRY.SIMPLETON_VALUE datatype was changed to LONGTEXT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.CUSTOM_LABELS datatype was changed to LONGTEXT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_DEFINITION datatype was changed to LONGTEXT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_INPUTS datatype was changed to LONGTEXT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_OPTIONS datatype was changed to LONGTEXT Jan 10, 2024 2:30:56 PM liquibase.changelog INFO: ChangeSet changesets/standardize_column_names_patches.xml::hsqldb_longvarchar_to_longtext::kshakir ran successfully in 18ms Running Changeset: changesets/nullable_lobs.xml::nullable_lobs::kshakir Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Null constraint dropped from CALL_CACHING_DETRITUS_ENTRY.DETRITUS_VALUE Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Null constraint dropped from CALL_CACHING_SIMPLETON_ENTRY.SIMPLETON_VALUE Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Null constraint dropped from JOB_STORE_SIMPLETON_ENTRY.SIMPLETON_VALUE Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_DEFINITION Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_INPUTS Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.WORKFLOW_OPTIONS Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Null constraint dropped from WORKFLOW_STORE_ENTRY.CUSTOM_LABELS Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/nullable_lobs.xml::nullable_lobs::kshakir ran successfully in 9ms Running Changeset: changesets/add_attempt_in_call_caching_entry.xml::add_attempt_in_call_caching_entry::tjeandet Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Columns JOB_ATTEMPT(INT) added to CALL_CACHING_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/add_attempt_in_call_caching_entry.xml::add_attempt_in_call_caching_entry::tjeandet ran successfully in 3ms Running Changeset: changesets/replace_empty_custom_labels.xml::custom_labels_not_null::rmunshi Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_STORE_ENTRY.CUSTOM_LABELS Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/replace_empty_custom_labels.xml::custom_labels_not_null::rmunshi ran successfully in 4ms Running Changeset: changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry::tjeandet Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table CALL_CACHING_AGGREGATION_ENTRY created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry::tjeandet ran successfully in 3ms Running Changeset: changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry_keys_and_indexes::tjeandet Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_AGGREGATION_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index IX_CALL_CACHING_AGGREGATION_ENTRY_BA_IFA created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry_keys_and_indexes::tjeandet ran successfully in 3ms Running Changeset: changesets/custom_label_entry.xml::custom_LABEL_entry::rmunshi Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::custom_LABEL_entry::rmunshi ran successfully in 3ms Running Changeset: changesets/custom_label_entry.xml::WMSE_workflow_execution_uuid_foreign_key::rmunshi Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Foreign key constraint added to CUSTOM_LABEL_ENTRY (WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::WMSE_workflow_execution_uuid_foreign_key::rmunshi ran successfully in 3ms Running Changeset: changesets/custom_label_entry.xml::custom_label_entry_index::rmunshi Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, CUSTOM_LABEL_VALUE, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::custom_label_entry_index::rmunshi ran successfully in 3ms Running Changeset: changesets/custom_label_entry.xml::modify label key/value data types::rmunshi Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_KEY datatype was changed to VARCHAR(63) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_VALUE datatype was changed to VARCHAR(63) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/custom_label_entry.xml::modify label key/value data types::rmunshi ran successfully in 4ms Running Changeset: changesets/docker_hash_store.xml::DOCKER_HASH_STORE_ENTRY::tjeandet Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table DOCKER_HASH_STORE_ENTRY created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store.xml::DOCKER_HASH_STORE_ENTRY::tjeandet ran successfully in 3ms Running Changeset: changesets/docker_hash_store.xml::docker_hash_store_uuid_unique_constraint::tjeandet Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Unique constraint added to DOCKER_HASH_STORE_ENTRY(WORKFLOW_EXECUTION_UUID, DOCKER_TAG) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store.xml::docker_hash_store_uuid_unique_constraint::tjeandet ran successfully in 3ms Running Changeset: changesets/workflow_store_type_and_version.xml::workflow-store-type-and-version::mcovarr Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Columns WORKFLOW_TYPE(VARCHAR(30)),WORKFLOW_TYPE_VERSION(VARCHAR(255)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_type_and_version.xml::workflow-store-type-and-version::mcovarr ran successfully in 4ms Running Changeset: changesets/remove_pre_pbe_tables.xml::drop-pre-pbe-tables::mcovarr Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table EXECUTION_INFO dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table EXECUTION_EVENT dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table FAILURE_EVENT dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table RUNTIME_ATTRIBUTES dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table SYMBOL dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table EXECUTION dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table WORKFLOW_EXECUTION_AUX dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table WORKFLOW_EXECUTION dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/remove_pre_pbe_tables.xml::drop-pre-pbe-tables::mcovarr ran successfully in 16ms Running Changeset: changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Foreign key FK_CUSTOM_LABEL_ENTRY_WORKFLOW_EXECUTION_UUID dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Unique constraint UC_CUSTOM_LABEL_ENTRY_CLK_CLV_WEU dropped from CUSTOM_LABEL_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Unique constraint UC_SUMMARY_STATUS_ENTRY_STN_STN dropped from SUMMARY_STATUS_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Unique constraint UC_WORKFLOW_METADATA_SUMMARY_ENTRY_WEU dropped from WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table METADATA_ENTRY dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table SUMMARY_STATUS_ENTRY dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY_ENTRY dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir ran successfully in 14ms Running Changeset: changesets/workflow_store_state_widening.xml::workflow-store-state-widening::tjeandet Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_STATE datatype was changed to varchar(20) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_state_widening.xml::workflow-store-state-widening::tjeandet ran successfully in 2ms Running Changeset: changesets/workflow_store_restarted_column.xml::workflow-store-restarted-column::tjeandet Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Columns RESTARTED(BOOLEAN) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_restarted_column.xml::workflow-store-restarted-column::tjeandet ran successfully in 4ms Running Changeset: changesets/workflow_store_restarted_column.xml::update-restartable::tjeandet Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Data updated in WORKFLOW_STORE_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Data updated in WORKFLOW_STORE_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_restarted_column.xml::update-restartable::tjeandet ran successfully in 3ms Running Changeset: changesets/workflow_store_workflow_root_column.xml::workflow-store-workflow-root-column::tjeandet Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Columns WORKFLOW_ROOT(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_workflow_root_column.xml::workflow-store-workflow-root-column::tjeandet ran successfully in 3ms Running Changeset: changesets/workflow_store_horizontal_db.xml::workflow-store-horizontal-db::mcovarr Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Column WORKFLOW_STORE_ENTRY.RESTARTED dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Columns CROMWELL_ID(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Columns HEARTBEAT_TIMESTAMP(TIMESTAMP) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/workflow_store_horizontal_db.xml::workflow-store-horizontal-db::mcovarr ran successfully in 5ms Running Changeset: changesets/add_workflow_url_in_workflow_store_entry.xml::add_workflow_url_in_workflow_store_entry::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Columns WORKFLOW_URL(VARCHAR(255)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/add_workflow_url_in_workflow_store_entry.xml::add_workflow_url_in_workflow_store_entry::sshah ran successfully in 3ms Running Changeset: changesets/change_max_size_workflow_url.xml::change_max_size_for_workflow_url::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.WORKFLOW_URL datatype was changed to VARCHAR(2000) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/change_max_size_workflow_url.xml::change_max_size_for_workflow_url::sshah ran successfully in 3ms Running Changeset: changesets/docker_hash_store_add_size_column.xml::docker_hash_store_entry_add_size_column::tjeandet Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Columns DOCKER_SIZE(BIGINT) added to DOCKER_HASH_STORE_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/docker_hash_store_add_size_column.xml::docker_hash_store_entry_add_size_column::tjeandet ran successfully in 3ms Running Changeset: changesets/enlarge_call_caching_hash_entry_id.xml::restore_auto_increment_call_caching_hash_entry_id::mcovarr Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_hash_entry_id.xml::restore_auto_increment_call_caching_hash_entry_id::mcovarr ran successfully in 3ms Running Changeset: changesets/add_hog_group_in_workflow_store.xml::add_hog_group_in_workflow_store::cjllanwarne Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Columns HOG_GROUP(VARCHAR(100)) added to WORKFLOW_STORE_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/add_hog_group_in_workflow_store.xml::add_hog_group_in_workflow_store::cjllanwarne ran successfully in 3ms Running Changeset: changesets/resync_engine_schema.xml::resync-engine-schema::kshakir Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.HEARTBEAT_TIMESTAMP datatype was changed to DATETIME(6) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: WORKFLOW_STORE_ENTRY.SUBMISSION_TIME datatype was changed to DATETIME(6) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: NOT NULL constraint has been added to WORKFLOW_STORE_ENTRY.SUBMISSION_TIME Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/resync_engine_schema.xml::resync-engine-schema::kshakir ran successfully in 5ms Running Changeset: changesets/enlarge_job_store_ids.xml::drop_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Foreign key FK_JOB_STORE_SIMPLETON_ENTRY_JOB_STORE_ENTRY_ID dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::drop_job_store_simpleton_entry_fk::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_entry_id::mcovarr Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_ENTRY.JOB_STORE_ENTRY_ID Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_entry_id::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_id::mcovarr Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Auto-increment added to JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_SIMPLETON_ENTRY_ID Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_id::mcovarr ran successfully in 2ms Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: JOB_STORE_SIMPLETON_ENTRY.JOB_STORE_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_fk::mcovarr ran successfully in 3ms Running Changeset: changesets/enlarge_job_store_ids.xml::recreate_job_store_simpleton_entry_fk::mcovarr Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Foreign key constraint added to JOB_STORE_SIMPLETON_ENTRY (JOB_STORE_ENTRY_ID) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_job_store_ids.xml::recreate_job_store_simpleton_entry_fk::mcovarr ran successfully in 3ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_AGGREGATION_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_detritus_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_DETRITUS_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_detritus_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_hash_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_HASH_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_hash_entry_call_caching_entry_id_fk::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Foreign key FK_CALL_CACHING_SIMPLETON_ENTRY_CALL_CACHING_ENTRY_ID dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::drop_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_cache_entry_id::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: CALL_CACHING_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_cache_entry_id::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::reset_call_cache_entry_id_autoincrement::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_ENTRY.CALL_CACHING_ENTRY_ID Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::reset_call_cache_entry_id_autoincrement::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_aggregation_entry_fk::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_aggregation_entry_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_detritus_entry_fk::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_detritus_entry_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_hash_entry_fk::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: CALL_CACHING_HASH_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_hash_entry_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_simpleton_entry_fk::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::enlarge_call_caching_simpleton_entry_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_AGGREGATION_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_aggregation_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_detritus_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_DETRITUS_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_detritus_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_hash_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_HASH_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_hash_entry_call_caching_entry_id_fk::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Foreign key constraint added to CALL_CACHING_SIMPLETON_ENTRY (CALL_CACHING_ENTRY_ID) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_cache_entry_ids.xml::recreate_call_caching_simpleton_entry_call_caching_entry_id_fk::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_caching_aggregation_entry_id.xml::enlarge_call_cache_aggregation_entry_id::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_AGGREGATION_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_aggregation_entry_id.xml::enlarge_call_cache_aggregation_entry_id::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_caching_aggregation_entry_id.xml::reset_call_caching_aggregation_entry_id_autoincrement::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_AGGREGATION_ENTRY.CALL_CACHING_AGGREGATION_ENTRY_ID Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_aggregation_entry_id.xml::reset_call_caching_aggregation_entry_id_autoincrement::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_caching_detritus_entry_id.xml::enlarge_call_cache_detritus_entry_id::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_DETRITUS_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_detritus_entry_id.xml::enlarge_call_cache_detritus_entry_id::sshah ran successfully in 1ms Running Changeset: changesets/enlarge_call_caching_detritus_entry_id.xml::reset_call_caching_detritus_entry_id_autoincrement::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_DETRITUS_ENTRY.CALL_CACHING_DETRITUS_ENTRY_ID Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_detritus_entry_id.xml::reset_call_caching_detritus_entry_id_autoincrement::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_simpleton_entry_id.xml::enlarge_call_cache_simpleton_entry_id::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_SIMPLETON_ENTRY_ID datatype was changed to BIGINT Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_simpleton_entry_id.xml::enlarge_call_cache_simpleton_entry_id::sshah ran successfully in 2ms Running Changeset: changesets/enlarge_call_caching_simpleton_entry_id.xml::reset_call_caching_simpleton_entry_id_autoincrement::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_SIMPLETON_ENTRY.CALL_CACHING_SIMPLETON_ENTRY_ID Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/enlarge_call_caching_simpleton_entry_id.xml::reset_call_caching_simpleton_entry_id_autoincrement::sshah ran successfully in 1ms Running Changeset: changesets/reset_call_caching_hash_entry_id_autoincrement.xml::reset_call_caching_hash_entry_id_autoincrement::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Auto-increment added to CALL_CACHING_HASH_ENTRY.CALL_CACHING_HASH_ENTRY_ID Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/reset_call_caching_hash_entry_id_autoincrement.xml::reset_call_caching_hash_entry_id_autoincrement::sshah ran successfully in 1ms Jan 10, 2024 2:30:57 PM liquibase.lockservice INFO: Successfully released change log lock [2024-01-10 14:30:57,32] [info] Running with database db.url = jdbc:hsqldb:mem:1e8982c5-6318-4a41-b0bc-328f95ec5313;shutdown=false;hsqldb.tx=mvcc Jan 10, 2024 2:30:57 PM liquibase.database INFO: Set default schema name to PUBLIC Jan 10, 2024 2:30:57 PM liquibase.lockservice INFO: Successfully acquired change log lock Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Creating database history table with name: PUBLIC.SQLMETADATADATABASECHANGELOG Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Reading from PUBLIC.SQLMETADATADATABASECHANGELOG Running Changeset: metadata_changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table CUSTOM_LABEL_ENTRY created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table METADATA_ENTRY created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table SUMMARY_STATUS_ENTRY created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table WORKFLOW_METADATA_SUMMARY_ENTRY created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, CUSTOM_LABEL_VALUE, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS_ENTRY(SUMMARY_TABLE_NAME, SUMMARIZED_TABLE_NAME) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Unique constraint added to WORKFLOW_METADATA_SUMMARY_ENTRY(WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Foreign key constraint added to CUSTOM_LABEL_ENTRY (WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WN created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_WS created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index METADATA_JOB_AND_KEY_IDX created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index METADATA_JOB_IDX created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index METADATA_WORKFLOW_IDX created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index SYS_IDX_11226 created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir ran successfully in 39ms Running Changeset: changesets/change_max_size_label_entry.xml::change_max_size_for_key_and_value_in_label::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_KEY datatype was changed to VARCHAR(255) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: CUSTOM_LABEL_ENTRY.CUSTOM_LABEL_VALUE datatype was changed to VARCHAR(255) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet changesets/change_max_size_label_entry.xml::change_max_size_for_key_and_value_in_label::sshah ran successfully in 3ms Running Changeset: metadata_changesets/delete_duplicate_custom_labels.xml::delete_duplicate_custom_labels::kshakir Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Data deleted from CUSTOM_LABEL_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Unique constraint UC_CUSTOM_LABEL_ENTRY_CLK_CLV_WEU dropped from CUSTOM_LABEL_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Unique constraint added to CUSTOM_LABEL_ENTRY(CUSTOM_LABEL_KEY, WORKFLOW_EXECUTION_UUID) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/delete_duplicate_custom_labels.xml::delete_duplicate_custom_labels::kshakir ran successfully in 7ms Running Changeset: metadata_changesets/add_submission_timestamp_metadata_summary.xml::add_submission_timestamp_in_metadata_entry::sshah Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Columns SUBMISSION_TIMESTAMP(DATETIME(6)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_submission_timestamp_metadata_summary.xml::add_submission_timestamp_in_metadata_entry::sshah ran successfully in 2ms Running Changeset: metadata_changesets/custom_label_entry_key_value_index.xml::custom_label_entry_key_value_index::mcovarr Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index IX_CUSTOM_LABEL_ENTRY_CLK_CLV created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/custom_label_entry_key_value_index.xml::custom_label_entry_key_value_index::mcovarr ran successfully in 3ms Running Changeset: metadata_changesets/add_parent_and_root_workflow_execution_uuid.xml::add_parent_and_root_workflow_execution_uuid::kshakir Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Columns PARENT_WORKFLOW_EXECUTION_UUID(VARCHAR(100)),ROOT_WORKFLOW_EXECUTION_UUID(VARCHAR(100)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_PWEU created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_RWEU created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Unique constraint UC_SUMMARY_STATUS_ENTRY_STN_STN dropped from SUMMARY_STATUS_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.SUMMARY_TABLE_NAME renamed to SUMMARY_NAME Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.SUMMARIZED_TABLE_NAME dropped Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Column SUMMARY_STATUS_ENTRY.MAXIMUM_ID renamed to SUMMARY_POSITION Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Unique constraint added to SUMMARY_STATUS_ENTRY(SUMMARY_NAME) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_parent_and_root_workflow_execution_uuid.xml::add_parent_and_root_workflow_execution_uuid::kshakir ran successfully in 16ms Running Changeset: metadata_changesets/add_parent_and_root_workflow_typo_fix.xml::add_parent_and_root_workflow_typo_fix::kshakir Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Data deleted from SUMMARY_STATUS_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Data updated in SUMMARY_STATUS_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_parent_and_root_workflow_typo_fix.xml::add_parent_and_root_workflow_typo_fix::kshakir ran successfully in 3ms Running Changeset: metadata_changesets/metadata_index_removals.xml::metadata_index_removals::mcovarr Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index METADATA_JOB_AND_KEY_IDX dropped from table METADATA_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index METADATA_JOB_IDX dropped from table METADATA_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/metadata_index_removals.xml::metadata_index_removals::mcovarr ran successfully in 2ms Running Changeset: metadata_changesets/add_metadata_archive_status.xml::add_metadata_archive_status::cjllanwarne Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Columns METADATA_ARCHIVE_STATUS(VARCHAR(30)) added to WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_metadata_archive_status.xml::add_metadata_archive_status::cjllanwarne ran successfully in 3ms Running Changeset: metadata_changesets/add_metadata_archive_status.xml::metadata_archive_status_index::cjllanwarne Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/add_metadata_archive_status.xml::metadata_archive_status_index::cjllanwarne ran successfully in 2ms Running Changeset: metadata_changesets/summarization_queue_table.xml::summary_queue_table::gsterin Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Table SUMMARY_QUEUE_ENTRY created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table.xml::summary_queue_table::gsterin ran successfully in 2ms Running Changeset: metadata_changesets/summarization_queue_table.xml::summary_queue_initialization::gsterin Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table.xml::summary_queue_initialization::gsterin ran successfully in 6ms Running Changeset: metadata_changesets/summarization_queue_table_add_primary_key.xml::summary_queue_add_primary_key::gsterin Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Primary key added to SUMMARY_QUEUE_ENTRY (METADATA_JOURNAL_ID) Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/summarization_queue_table_add_primary_key.xml::summary_queue_add_primary_key::gsterin ran successfully in 2ms Running Changeset: metadata_changesets/remove_non_summarizable_metadata_from_queue.xml::delete_non_summarizable_metadata_from_queue::mcovarr Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Custom SQL executed Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/remove_non_summarizable_metadata_from_queue.xml::delete_non_summarizable_metadata_from_queue::mcovarr ran successfully in 4ms Running Changeset: metadata_changesets/update_metadata_archive_index.xml::update_old_metadata_archive_status_index::cjllanwarne Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS dropped from table WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Index IX_WORKFLOW_METADATA_SUMMARY_ENTRY_MAS_ET created Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/update_metadata_archive_index.xml::update_old_metadata_archive_status_index::cjllanwarne ran successfully in 3ms Running Changeset: metadata_changesets/reset_archive_statuses_to_null.xml::reset_archive_statuses_to_null::cjllanwarne Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: Data updated in WORKFLOW_METADATA_SUMMARY_ENTRY Jan 10, 2024 2:30:57 PM liquibase.changelog INFO: ChangeSet metadata_changesets/reset_archive_statuses_to_null.xml::reset_archive_statuses_to_null::cjllanwarne ran successfully in 2ms Jan 10, 2024 2:30:57 PM liquibase.lockservice INFO: Successfully released change log lock [2024-01-10 14:30:58,04] [info] Slf4jLogger started [2024-01-10 14:30:58,36] [info] Workflow heartbeat configuration: { "cromwellId" : "cromid-c27a428", "heartbeatInterval" : "2 minutes", "ttl" : "10 minutes", "failureShutdownDuration" : "5 minutes", "writeBatchSize" : 10000, "writeThreshold" : 10000 } [2024-01-10 14:30:58,46] [info] Metadata summary refreshing every 1 second. [2024-01-10 14:30:58,46] [info] No metadata archiver defined in config [2024-01-10 14:30:58,46] [info] No metadata deleter defined in config [2024-01-10 14:30:58,49] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds. [2024-01-10 14:30:58,49] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds. [2024-01-10 14:30:58,53] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds. [2024-01-10 14:30:58,61] [info] JobRestartCheckTokenDispenser - Distribution rate: 50 per 1 seconds. [2024-01-10 14:30:58,65] [info] JobExecutionTokenDispenser - Distribution rate: 20 per 10 seconds. [2024-01-10 14:30:58,72] [info] SingleWorkflowRunnerActor: Version 83 [2024-01-10 14:30:58,74] [info] SingleWorkflowRunnerActor: Submitting workflow [2024-01-10 14:30:58,80] [info] Unspecified type (Unspecified version) workflow a74105f8-2597-4528-bb3c-9eb852973d92 submitted [2024-01-10 14:30:58,83] [info] SingleWorkflowRunnerActor: Workflow submitted a74105f8-2597-4528-bb3c-9eb852973d92 [2024-01-10 14:30:58,84] [info] 1 new workflows fetched by cromid-c27a428: a74105f8-2597-4528-bb3c-9eb852973d92 [2024-01-10 14:30:58,85] [info] WorkflowManagerActor: Starting workflow a74105f8-2597-4528-bb3c-9eb852973d92 [2024-01-10 14:30:58,86] [info] WorkflowManagerActor: Successfully started WorkflowActor-a74105f8-2597-4528-bb3c-9eb852973d92 [2024-01-10 14:30:58,86] [info] Retrieved 1 workflows from the WorkflowStoreActor [2024-01-10 14:30:58,90] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes. [2024-01-10 14:30:59,03] [info] MaterializeWorkflowDescriptorActor [a74105f8]: Parsing workflow as WDL 1.0 [2024-01-10 14:31:01,23] [info] MaterializeWorkflowDescriptorActor [a74105f8]: Call-to-Backend assignments: theiacov_ont.nextclade_flu_na -> Local, theiacov_ont.clean_check_reads -> Local, theiacov_ont.vadr -> Local, theiacov_ont.nextclade_output_parser_flu_na -> Local, read_QC_trim_ont.kmc -> Local, theiacov_ont.sc2_gene_coverage -> Local, theiacov_ont.abricate_flu -> Local, theiacov_ont.irma -> Local, theiacov_ont.consensus_qc -> Local, theiacov_ont.version_capture -> Local, theiacov_ont.nanoplot_clean -> Local, theiacov_ont.raw_check_reads -> Local, theiacov_ont.nanoplot_raw -> Local, read_QC_trim_ont.nanoq -> Local, read_QC_trim_ont.tiptoft -> Local, read_QC_trim_ont.rasusa -> Local, theiacov_ont.stats_n_coverage_primtrim -> Local, read_QC_trim_ont.read_filtering -> Local, theiacov_ont.stats_n_coverage -> Local, read_QC_trim_ont.ncbi_scrub_se -> Local, theiacov_ont.consensus -> Local, theiacov_ont.qc_check_task -> Local, theiacov_ont.nextclade -> Local, theiacov_ont.pangolin4 -> Local, theiacov_ont.quasitools_ont -> Local, read_QC_trim_ont.kraken2_dehosted -> Local, theiacov_ont.nextclade_output_parser -> Local, read_QC_trim_ont.kraken2_raw -> Local [2024-01-10 14:31:01,48] [warn] Local [a74105f8]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,48] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,48] [warn] Local [a74105f8]: Key/s [dx_instance_type, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [disk, dx_instance_type, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:01,49] [warn] Local [a74105f8]: Key/s [preemptible, disk, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2024-01-10 14:31:02,61] [info] WorkflowExecutionActor-a74105f8-2597-4528-bb3c-9eb852973d92 [a74105f8]: Starting theiacov_ont.version_capture [2024-01-10 14:31:03,62] [info] Not triggering log of restart checking token queue status. Effective log interval = None [2024-01-10 14:31:03,67] [info] Not triggering log of execution token queue status. Effective log interval = None [2024-01-10 14:31:03,69] [info] WorkflowExecutionActor-a74105f8-2597-4528-bb3c-9eb852973d92 [a74105f8]: Starting theiacov_ont.raw_check_reads [2024-01-10 14:31:08,68] [info] Assigned new job execution tokens to the following groups: a74105f8: 2 [2024-01-10 14:31:09,72] [warn] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.version_capture:NA:1]: Unrecognized runtime attribute keys: dx_instance_type, disks, cpu, memory [2024-01-10 14:31:09,72] [warn] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.raw_check_reads:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:31:09,83] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.version_capture:NA:1]: PHB_Version="PHB v1.2.1" date +"%Y-%m-%d" > TODAY echo "$PHB_Version" > PHB_VERSION [2024-01-10 14:31:09,84] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.raw_check_reads:NA:1]: flag="PASS" # initalize estimated genome size estimated_genome_size=0 if [[ "false" == "false" ]] ; then # set cat command based on compression if [[ "/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-raw_check_reads/inputs/1032112200/ont.fastq.gz" == *".gz" ]] ; then cat_reads="zcat" else cat_reads="cat" fi # check one: number of reads read1_num=eval "$cat_reads /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-raw_check_reads/inputs/1032112200/ont.fastq.gz" | awk '{s++}END{print s/4}' # awk '{s++}END{print s/4' counts the number of lines and divides them by 4 # key assumption: in fastq there will be four lines per read # sometimes fastqs do not have 4 lines per read, so this might fail one day if [ "${read1_num}" -le "113" ] ; then flag="FAIL; the number of reads (${read1_num}) is below the minimum of 113" else flag="PASS" fi # checks two and three: number of basepairs and proportion of sequence if [ "${flag}" == "PASS" ]; then # count number of basepairs # this only works if the fastq has 4 lines per read, so this might fail one day read1_bp=eval "${cat_reads} /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-raw_check_reads/inputs/1032112200/ont.fastq.gz" | paste - - - - | cut -f2 | tr -d '\n' | wc -c # paste - - - - (print 4 consecutive lines in one row, tab delimited) # cut -f2 print only the second column (the second line of the fastq 4-line) # tr -d '\n' removes line endings # wc -c counts characters if [ "$flag" == "PASS" ] ; then if [ "${read1_bp}" -le "34000" ] ; then flag="FAIL; the number of basepairs (${read1_bp}) is below the minimum of 34000" else flag="PASS" fi fi fi #checks four and five: estimated genome size and coverage if [ "${flag}" == "PASS" ] && [ "false" == "false" ]; then # estimate genome size if theiaprok AND expected_genome_size was not provided if [ "theiacov" == "theiaprok" ] && [[ -z "" ]]; then # First Pass; assuming average depth mash sketch -o test -k 31 -m 3 -r /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-raw_check_reads/inputs/1032112200/ont.fastq.gz > mash-output.txt 2>&1 grep "Estimated genome size:" mash-output.txt | \ awk '{if($4){printf("%d", $4)}} END {if (!NR) print "0"}' > genome_size_output grep "Estimated coverage:" mash-output.txt | \ awk '{if($3){printf("%d", $3)}} END {if (!NR) print "0"}' > coverage_output # remove mash outputs rm -rf test.msh rm -rf mash-output.txt estimated_genome_size=head -n1 genome_size_output estimated_coverage=head -n1 coverage_output # Check if second pass is needed if [ ${estimated_genome_size} -gt "2673870" ] || [ ${estimated_genome_size} -lt "1700" ] ; then # Probably high coverage, try increasing number of kmer copies to 10 M="-m 10" if [ ${estimated_genome_size} -lt "1700" ]; then # Probably low coverage, try decreasing the number of kmer copies to 1 M="-m 1" fi mash sketch -o test -k 31 ${M} -r /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-raw_check_reads/inputs/1032112200/ont.fastq.gz > mash-output.txt 2>&1 grep "Estimated genome size:" mash-output.txt | \ awk '{if($4){printf("%d", $4)}} END {if (!NR) print "0"}' > genome_size_output grep "Estimated coverage:" mash-output.txt | \ awk '{if($3){printf("%d", $3)}} END {if (!NR) print "0"}' > coverage_output # remove mash outputs rm -rf test.msh rm -rf mash-output.txt fi estimated_genome_size=head -n1 genome_size_output estimated_coverage=head -n1 coverage_output # estimate coverage if theiacov OR expected_genome_size was provided elif [ "theiacov" == "theiacov" ] || [ "" ]; then if [ "" ]; then estimated_genome_size= # use user-provided expected_genome_size elif [ "sars-cov-2" == "sars-cov-2" ]; then estimated_genome_size=29903 # size taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_045512.2 elif [ "sars-cov-2" == "MPXV" ]; then estimated_genome_size=197209 # size of 2022 virus taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_063383.1 elif [ "sars-cov-2" == "flu" ]; then estimated_genome_size=14000 # 500 bp over the CDC's approximate full genome size of 13500 (see https://www.cdc.gov/flu/about/professionals/genetic-characterization.htm) elif [ "sars-cov-2" == "HIV" ]; then estimated_genome_size=9181 # size taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_001802.1 elif [ "sars-cov-2" == "WNV" ]; then estimated_genome_size=11092 # WNV lineage 1 size from https://www.ncbi.nlm.nih.gov/nuccore/NC_009942.1 else flag="FAIL; the organism tag provided (sars-cov-2) is not valid and no expected_genome_size was provided." fi # coverage is calculated here by N/G where N is number of bases, and G is genome length # this will nearly always be an overestimation if [ $estimated_genome_size -ne 0 ]; then # prevent divided by zero errors estimated_coverage=$(python3 -c "print(round(($read1_bp)/$estimated_genome_size))") else # they provided 0 for estimated_genome_size, nice estimated_coverage=0 fi else # workflow series was not provided; default to fail estimated_genome_size=0 estimated_coverage=0 fi if [ "${estimated_genome_size}" -ge "2673870" ] ; then flag="FAIL; the estimated genome size (${estimated_genome_size}) is larger than the maximum of 2673870 bps" elif [ "${estimated_genome_size}" -le "1700" ] ; then flag="FAIL; the estimated genome size (${estimated_genome_size}) is smaller than the minimum of 1700 bps" else flag="PASS" if [ "${estimated_coverage}" -lt "10" ] ; then flag="FAIL; the estimated coverage (${estimated_coverage}) is less than the minimum of 10x" else flag="PASS" echo $estimated_genome_size | tee EST_GENOME_LENGTH fi fi fi fi echo $flag | tee FLAG echo $estimated_genome_size | tee EST_GENOME_LENGTH [2024-01-10 14:31:09,99] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.version_capture:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-version_capture/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-version_capture/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-version_capture:/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-version_capture:delegated \ us-docker.pkg.dev/general-theiagen/ubuntu/ubuntu@sha256:9a35cef02b57290a7c919824b632897fbe6db3aa18c930110b04d81fcad458ff /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-version_capture/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-version_capture/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-version_capture/execution/docker_cid # return exit code exit $rc [2024-01-10 14:31:09,99] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.raw_check_reads:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-raw_check_reads/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-raw_check_reads/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-raw_check_reads:/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-raw_check_reads:delegated \ us-docker.pkg.dev/general-theiagen/bactopia/gather_samples@sha256:2e044c7be68519b9d683cd242177518e6db662b40b3bf669f891954f90a4d9a5 /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-raw_check_reads/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-raw_check_reads/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-raw_check_reads/execution/docker_cid # return exit code exit $rc [2024-01-10 14:31:13,54] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.raw_check_reads:NA:1]: job id: 62270 [2024-01-10 14:31:13,54] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.version_capture:NA:1]: job id: 62269 [2024-01-10 14:31:13,55] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.raw_check_reads:NA:1]: Status change from - to Done [2024-01-10 14:31:13,55] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.version_capture:NA:1]: Status change from - to Done [2024-01-10 14:31:21,06] [info] dbaaa31c-b302-4d0d-9600-5d4ddf59c076-SubWorkflowActor-SubWorkflow-read_qc_trim:-1:1 [dbaaa31c]: Starting read_QC_trim_ont.ncbi_scrub_se, read_QC_trim_ont.kraken2_raw [2024-01-10 14:31:28,67] [info] Assigned new job execution tokens to the following groups: a74105f8: 2 [2024-01-10 14:31:28,89] [warn] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.kraken2_raw:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:31:28,91] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.kraken2_raw:NA:1]: # date and version control date | tee DATE kraken2 --version | head -n1 | tee VERSION num_reads=$(ls *fastq.gz 2> /dev/nul | wc -l) if ! [ -z ]; then mode="--paired" fi echo $mode kraken2 $mode \ --threads 4 \ --db /kraken2-db \ /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_raw/inputs/1032112200/ont.fastq.gz \ --report ont_kraken2_report.txt >/dev/null percentage_human=$(grep "Homo sapiens" ont_kraken2_report.txt | cut -f 1) # | tee PERCENT_HUMAN percentage_sc2=$(grep "Severe acute respiratory syndrome coronavirus 2" ont_kraken2_report.txt | cut -f1 ) # | tee PERCENT_COV if [ -z "$percentage_human" ] ; then percentage_human="0" ; fi if [ -z "$percentage_sc2" ] ; then percentage_sc2="0" ; fi echo $percentage_human | tee PERCENT_HUMAN echo $percentage_sc2 | tee PERCENT_SC2 # capture target org percentage if [ ! -z "" ]; then echo "Target org designated: " percent_target_org=$(grep "" ont_kraken2_report.txt | cut -f1 | head -n1 ) if [-z "$percent_target_org" ] ; then percent_target_org="0" ; fi else percent_target_org="" fi echo $percent_target_org | tee PERCENT_TARGET_ORG [2024-01-10 14:31:28,93] [warn] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.ncbi_scrub_se:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:31:28,94] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.ncbi_scrub_se:NA:1]: # date and version control date | tee DATE # unzip fwd file as scrub tool does not take in .gz fastq files if [[ "/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-ncbi_scrub_se/inputs/1032112200/ont.fastq.gz" == *.gz ]] then gunzip -c /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-ncbi_scrub_se/inputs/1032112200/ont.fastq.gz > r1.fastq read1_unzip=r1.fastq else read1_unzip=/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-ncbi_scrub_se/inputs/1032112200/ont.fastq.gz fi # dehost reads /opt/scrubber/scripts/scrub.sh -n ${read1_unzip} |& tail -n1 | awk -F" " '{print $1}' > FWD_SPOTS_REMOVED # gzip dehosted reads gzip ${read1_unzip}.clean -c > ont_R1_dehosted.fastq.gz [2024-01-10 14:31:28,96] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.ncbi_scrub_se:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-ncbi_scrub_se/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-ncbi_scrub_se/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-ncbi_scrub_se:/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-ncbi_scrub_se:delegated \ us-docker.pkg.dev/general-theiagen/ncbi/sra-human-scrubber@sha256:b7dba71079344daea4ea3363e1a67fa54edb7ec65459d039669c68a66d38b140 /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-ncbi_scrub_se/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-ncbi_scrub_se/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-ncbi_scrub_se/execution/docker_cid # return exit code exit $rc [2024-01-10 14:31:28,96] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.kraken2_raw:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_raw/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_raw/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_raw:/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_raw:delegated \ us-docker.pkg.dev/general-theiagen/staphb/kraken2@sha256:7884b5a6ec3c20cd7edbd42b92ae7e9806b4581997acb270b1b9ecb627474e2a /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_raw/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_raw/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_raw/execution/docker_cid # return exit code exit $rc [2024-01-10 14:31:33,51] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.kraken2_raw:NA:1]: job id: 62686 [2024-01-10 14:31:33,51] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.ncbi_scrub_se:NA:1]: job id: 62684 [2024-01-10 14:31:33,51] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.kraken2_raw:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:31:33,51] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.ncbi_scrub_se:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:31:41,48] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.ncbi_scrub_se:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:31:43,50] [info] dbaaa31c-b302-4d0d-9600-5d4ddf59c076-SubWorkflowActor-SubWorkflow-read_qc_trim:-1:1 [dbaaa31c]: Starting read_QC_trim_ont.read_filtering, read_QC_trim_ont.kraken2_dehosted [2024-01-10 14:31:48,66] [info] Assigned new job execution tokens to the following groups: a74105f8: 2 [2024-01-10 14:31:48,68] [warn] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.kraken2_dehosted:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:31:48,68] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_dehosted/inputs/-1729794163/ont_R1_dehosted.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-ncbi_scrub_se/execution/ont_R1_dehosted.fastq.gz: Operation not permitted [2024-01-10 14:31:48,71] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.kraken2_dehosted:NA:1]: # date and version control date | tee DATE kraken2 --version | head -n1 | tee VERSION num_reads=$(ls *fastq.gz 2> /dev/nul | wc -l) if ! [ -z ]; then mode="--paired" fi echo $mode kraken2 $mode \ --threads 4 \ --db /kraken2-db \ /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_dehosted/inputs/-1729794163/ont_R1_dehosted.fastq.gz \ --report ont_kraken2_report.txt >/dev/null percentage_human=$(grep "Homo sapiens" ont_kraken2_report.txt | cut -f 1) # | tee PERCENT_HUMAN percentage_sc2=$(grep "Severe acute respiratory syndrome coronavirus 2" ont_kraken2_report.txt | cut -f1 ) # | tee PERCENT_COV if [ -z "$percentage_human" ] ; then percentage_human="0" ; fi if [ -z "$percentage_sc2" ] ; then percentage_sc2="0" ; fi echo $percentage_human | tee PERCENT_HUMAN echo $percentage_sc2 | tee PERCENT_SC2 # capture target org percentage if [ ! -z "" ]; then echo "Target org designated: " percent_target_org=$(grep "" ont_kraken2_report.txt | cut -f1 | head -n1 ) if [-z "$percent_target_org" ] ; then percent_target_org="0" ; fi else percent_target_org="" fi echo $percent_target_org | tee PERCENT_TARGET_ORG [2024-01-10 14:31:48,75] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.kraken2_dehosted:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_dehosted/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_dehosted/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_dehosted:/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_dehosted:delegated \ us-docker.pkg.dev/general-theiagen/staphb/kraken2@sha256:7884b5a6ec3c20cd7edbd42b92ae7e9806b4581997acb270b1b9ecb627474e2a /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_dehosted/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_dehosted/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_dehosted/execution/docker_cid # return exit code exit $rc [2024-01-10 14:31:48,88] [warn] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.read_filtering:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:31:48,89] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-read_filtering/inputs/-1729794163/ont_R1_dehosted.fastq.gz -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-ncbi_scrub_se/execution/ont_R1_dehosted.fastq.gz: Operation not permitted [2024-01-10 14:31:48,93] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.read_filtering:NA:1]: # date and version control mkdir ont cp /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-read_filtering/inputs/-1729794163/ont_R1_dehosted.fastq.gz ont/ echo "DIRNAME: $(dirname)" # run artic guppyplex artic guppyplex --min-length 400 --max-length 700 --directory ont --prefix artic_ncov2019 [2024-01-10 14:31:48,93] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.read_filtering:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-read_filtering/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-read_filtering/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-read_filtering:/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-read_filtering:delegated \ us-docker.pkg.dev/general-theiagen/staphb/artic-ncov2019@sha256:ee7b325330acc5cb738efcdb267d558ac0ca6a03ba22dc46393c3168a5469b56 /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-read_filtering/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-read_filtering/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-read_filtering/execution/docker_cid # return exit code exit $rc [2024-01-10 14:31:50,96] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.kraken2_raw:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:31:53,51] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.read_filtering:NA:1]: job id: 63080 [2024-01-10 14:31:53,51] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.kraken2_dehosted:NA:1]: job id: 63055 [2024-01-10 14:31:53,51] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.kraken2_dehosted:NA:1]: Status change from - to Done [2024-01-10 14:31:53,51] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.read_filtering:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:32:02,61] [info] BackgroundConfigAsyncJobExecutionActor [dbaaa31cread_QC_trim_ont.read_filtering:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:32:04,96] [info] dbaaa31c-b302-4d0d-9600-5d4ddf59c076-SubWorkflowActor-SubWorkflow-read_qc_trim:-1:1 [dbaaa31c]: Workflow read_QC_trim_ont complete. Final Outputs: { "read_QC_trim_ont.kraken_report": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_raw/execution/ont_kraken2_report.txt", "read_QC_trim_ont.tiptoft_plasmid_replicon_genes": null, "read_QC_trim_ont.est_genome_size": null, "read_QC_trim_ont.kraken_human_dehosted": 0.05, "read_QC_trim_ont.kraken_sc2": 99.41, "read_QC_trim_ont.kmc_version": null, "read_QC_trim_ont.kraken_human": 0.43, "read_QC_trim_ont.kraken_version": "Kraken version 2.0.8-beta", "read_QC_trim_ont.tiptoft_version": null, "read_QC_trim_ont.kraken_target_org_dehosted": "", "read_QC_trim_ont.read1_dehosted": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-ncbi_scrub_se/execution/ont_R1_dehosted.fastq.gz", "read_QC_trim_ont.kraken_target_org_name": null, "read_QC_trim_ont.rasusa_version": null, "read_QC_trim_ont.tiptoft_plasmid_replicon_fastq": null, "read_QC_trim_ont.kraken_report_dehosted": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-kraken2_dehosted/execution/ont_kraken2_report.txt", "read_QC_trim_ont.tiptoft_result_tsv": null, "read_QC_trim_ont.nanoq_version": null, "read_QC_trim_ont.kraken_target_org": "", "read_QC_trim_ont.read1_clean": "/home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-read_filtering/execution/artic_ncov2019_ont.fastq", "read_QC_trim_ont.kmc_kmer_stats": null, "read_QC_trim_ont.kraken_sc2_dehosted": 99.4 } [2024-01-10 14:32:09,93] [info] WorkflowExecutionActor-a74105f8-2597-4528-bb3c-9eb852973d92 [a74105f8]: Starting theiacov_ont.clean_check_reads [2024-01-10 14:32:18,67] [info] Assigned new job execution tokens to the following groups: a74105f8: 1 [2024-01-10 14:32:18,68] [warn] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.clean_check_reads:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:32:18,69] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-clean_check_reads/inputs/1986200811/artic_ncov2019_ont.fastq -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-read_filtering/execution/artic_ncov2019_ont.fastq: Operation not permitted [2024-01-10 14:32:18,71] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.clean_check_reads:NA:1]: flag="PASS" # initalize estimated genome size estimated_genome_size=0 if [[ "false" == "false" ]] ; then # set cat command based on compression if [[ "/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-clean_check_reads/inputs/1986200811/artic_ncov2019_ont.fastq" == *".gz" ]] ; then cat_reads="zcat" else cat_reads="cat" fi # check one: number of reads read1_num=eval "$cat_reads /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-clean_check_reads/inputs/1986200811/artic_ncov2019_ont.fastq" | awk '{s++}END{print s/4}' # awk '{s++}END{print s/4' counts the number of lines and divides them by 4 # key assumption: in fastq there will be four lines per read # sometimes fastqs do not have 4 lines per read, so this might fail one day if [ "${read1_num}" -le "113" ] ; then flag="FAIL; the number of reads (${read1_num}) is below the minimum of 113" else flag="PASS" fi # checks two and three: number of basepairs and proportion of sequence if [ "${flag}" == "PASS" ]; then # count number of basepairs # this only works if the fastq has 4 lines per read, so this might fail one day read1_bp=eval "${cat_reads} /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-clean_check_reads/inputs/1986200811/artic_ncov2019_ont.fastq" | paste - - - - | cut -f2 | tr -d '\n' | wc -c # paste - - - - (print 4 consecutive lines in one row, tab delimited) # cut -f2 print only the second column (the second line of the fastq 4-line) # tr -d '\n' removes line endings # wc -c counts characters if [ "$flag" == "PASS" ] ; then if [ "${read1_bp}" -le "34000" ] ; then flag="FAIL; the number of basepairs (${read1_bp}) is below the minimum of 34000" else flag="PASS" fi fi fi #checks four and five: estimated genome size and coverage if [ "${flag}" == "PASS" ] && [ "false" == "false" ]; then # estimate genome size if theiaprok AND expected_genome_size was not provided if [ "theiacov" == "theiaprok" ] && [[ -z "" ]]; then # First Pass; assuming average depth mash sketch -o test -k 31 -m 3 -r /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-clean_check_reads/inputs/1986200811/artic_ncov2019_ont.fastq > mash-output.txt 2>&1 grep "Estimated genome size:" mash-output.txt | \ awk '{if($4){printf("%d", $4)}} END {if (!NR) print "0"}' > genome_size_output grep "Estimated coverage:" mash-output.txt | \ awk '{if($3){printf("%d", $3)}} END {if (!NR) print "0"}' > coverage_output # remove mash outputs rm -rf test.msh rm -rf mash-output.txt estimated_genome_size=head -n1 genome_size_output estimated_coverage=head -n1 coverage_output # Check if second pass is needed if [ ${estimated_genome_size} -gt "2673870" ] || [ ${estimated_genome_size} -lt "1700" ] ; then # Probably high coverage, try increasing number of kmer copies to 10 M="-m 10" if [ ${estimated_genome_size} -lt "1700" ]; then # Probably low coverage, try decreasing the number of kmer copies to 1 M="-m 1" fi mash sketch -o test -k 31 ${M} -r /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-clean_check_reads/inputs/1986200811/artic_ncov2019_ont.fastq > mash-output.txt 2>&1 grep "Estimated genome size:" mash-output.txt | \ awk '{if($4){printf("%d", $4)}} END {if (!NR) print "0"}' > genome_size_output grep "Estimated coverage:" mash-output.txt | \ awk '{if($3){printf("%d", $3)}} END {if (!NR) print "0"}' > coverage_output # remove mash outputs rm -rf test.msh rm -rf mash-output.txt fi estimated_genome_size=head -n1 genome_size_output estimated_coverage=head -n1 coverage_output # estimate coverage if theiacov OR expected_genome_size was provided elif [ "theiacov" == "theiacov" ] || [ "" ]; then if [ "" ]; then estimated_genome_size= # use user-provided expected_genome_size elif [ "sars-cov-2" == "sars-cov-2" ]; then estimated_genome_size=29903 # size taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_045512.2 elif [ "sars-cov-2" == "MPXV" ]; then estimated_genome_size=197209 # size of 2022 virus taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_063383.1 elif [ "sars-cov-2" == "flu" ]; then estimated_genome_size=14000 # 500 bp over the CDC's approximate full genome size of 13500 (see https://www.cdc.gov/flu/about/professionals/genetic-characterization.htm) elif [ "sars-cov-2" == "HIV" ]; then estimated_genome_size=9181 # size taken from https://www.ncbi.nlm.nih.gov/nuccore/NC_001802.1 elif [ "sars-cov-2" == "WNV" ]; then estimated_genome_size=11092 # WNV lineage 1 size from https://www.ncbi.nlm.nih.gov/nuccore/NC_009942.1 else flag="FAIL; the organism tag provided (sars-cov-2) is not valid and no expected_genome_size was provided." fi # coverage is calculated here by N/G where N is number of bases, and G is genome length # this will nearly always be an overestimation if [ $estimated_genome_size -ne 0 ]; then # prevent divided by zero errors estimated_coverage=$(python3 -c "print(round(($read1_bp)/$estimated_genome_size))") else # they provided 0 for estimated_genome_size, nice estimated_coverage=0 fi else # workflow series was not provided; default to fail estimated_genome_size=0 estimated_coverage=0 fi if [ "${estimated_genome_size}" -ge "2673870" ] ; then flag="FAIL; the estimated genome size (${estimated_genome_size}) is larger than the maximum of 2673870 bps" elif [ "${estimated_genome_size}" -le "1700" ] ; then flag="FAIL; the estimated genome size (${estimated_genome_size}) is smaller than the minimum of 1700 bps" else flag="PASS" if [ "${estimated_coverage}" -lt "10" ] ; then flag="FAIL; the estimated coverage (${estimated_coverage}) is less than the minimum of 10x" else flag="PASS" echo $estimated_genome_size | tee EST_GENOME_LENGTH fi fi fi fi echo $flag | tee FLAG echo $estimated_genome_size | tee EST_GENOME_LENGTH [2024-01-10 14:32:18,73] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.clean_check_reads:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-clean_check_reads/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-clean_check_reads/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-clean_check_reads:/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-clean_check_reads:delegated \ us-docker.pkg.dev/general-theiagen/bactopia/gather_samples@sha256:2e044c7be68519b9d683cd242177518e6db662b40b3bf669f891954f90a4d9a5 /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-clean_check_reads/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-clean_check_reads/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-clean_check_reads/execution/docker_cid # return exit code exit $rc [2024-01-10 14:32:23,51] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.clean_check_reads:NA:1]: job id: 63603 [2024-01-10 14:32:23,51] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.clean_check_reads:NA:1]: Status change from - to Done [2024-01-10 14:32:29,31] [info] WorkflowExecutionActor-a74105f8-2597-4528-bb3c-9eb852973d92 [a74105f8]: Starting theiacov_ont.consensus [2024-01-10 14:32:38,67] [info] Assigned new job execution tokens to the following groups: a74105f8: 1 [2024-01-10 14:32:38,91] [warn] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.consensus:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:32:38,92] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/inputs/1986200811/artic_ncov2019_ont.fastq -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-read_filtering/execution/artic_ncov2019_ont.fastq: Operation not permitted [2024-01-10 14:32:38,94] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.consensus:NA:1]: # HIV if [[ sars-cov-2 == "HIV" ]]; then # setup custom primer scheme (/V is required by Artic) mkdir -p ./primer-schemes/HIV/Vuser ## set reference genome ref_genome="" head -n1 "${ref_genome}" | sed 's/>//' | tee REFERENCE_GENOME cp "${ref_genome}" ./primer-schemes/HIV/Vuser/HIV.reference.fasta ## set primers #cp /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/inputs/1063836572/artic-v3.primers.bed ./primer-schemes/SARS-CoV-2/Vuser/SARS-CoV-2.scheme.bed #p_bed="/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/inputs/1063836572/artic-v3.primers.bed" cp "/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/inputs/1063836572/artic-v3.primers.bed" ./primer-schemes/HIV/Vuser/HIV.scheme.bed scheme_name="HIV/Vuser" # Add other viruses here # Default is SARS-CoV-2 else # setup custom primer scheme (/V is required by Artic) mkdir -p ./primer-schemes/SARS-CoV-2/Vuser ## set reference genome if [[ ! -z "" ]]; then ref_genome="" else # use reference file in docker--different paths depending on image specified if [[ -d "/fieldbioinformatics" ]]; then ref_genome=$(find /fieldbioinformatics/*/primer*schemes/nCoV-2019/V3/ -name "nCoV-2019.reference.fasta") else ref_genome=$(find /wf-artic*/data/primer_schemes/SARS-CoV-2/V4/ -name "SARS-CoV-2.reference.fasta") fi echo "No user-defined reference genome; setting reference to ${ref_genome}" fi head -n1 "${ref_genome}" | sed 's/>//' | tee REFERENCE_GENOME cp "${ref_genome}" ./primer-schemes/SARS-CoV-2/Vuser/SARS-CoV-2.reference.fasta ## set primers cp /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/inputs/1063836572/artic-v3.primers.bed ./primer-schemes/SARS-CoV-2/Vuser/SARS-CoV-2.scheme.bed scheme_name="SARS-CoV-2/Vuser" fi # version control echo "Medaka via $(artic -v)" | tee VERSION echo "artic-v3.primers.bed" | tee PRIMER_NAME artic minion --medaka --medaka-model r941_min_high_g360 --normalise 200 --threads 8 --scheme-directory ./primer-schemes --read-file /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/inputs/1986200811/artic_ncov2019_ont.fastq ${scheme_name} ont gunzip -f ont.pass.vcf.gz # clean up fasta header echo ">ont" > ont.medaka.consensus.fasta grep -v ">" ont.consensus.fasta >> ont.medaka.consensus.fasta # grab reads from alignment samtools fastq -F4 ont.primertrimmed.rg.sorted.bam | gzip > ont.fastq.gz [2024-01-10 14:32:38,96] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.consensus:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus:/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus:delegated \ us-docker.pkg.dev/general-theiagen/staphb/artic-ncov2019-epi2me@sha256:678c5f3ebdaca0d736a4e59685b6214d455ab0964e88bdcecce07c99b77d5e4b /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/execution/docker_cid # return exit code exit $rc [2024-01-10 14:32:43,51] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.consensus:NA:1]: job id: 63871 [2024-01-10 14:32:43,52] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.consensus:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:37:50,88] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.consensus:NA:1]: Status change from WaitingForReturnCode to Done [2024-01-10 14:37:52,05] [info] WorkflowExecutionActor-a74105f8-2597-4528-bb3c-9eb852973d92 [a74105f8]: Starting theiacov_ont.stats_n_coverage_primtrim, theiacov_ont.stats_n_coverage [2024-01-10 14:37:53,07] [info] WorkflowExecutionActor-a74105f8-2597-4528-bb3c-9eb852973d92 [a74105f8]: Starting theiacov_ont.nextclade, theiacov_ont.sc2_gene_coverage, theiacov_ont.pangolin4, theiacov_ont.consensus_qc [2024-01-10 14:37:58,67] [info] Assigned new job execution tokens to the following groups: a74105f8: 6 [2024-01-10 14:37:58,96] [warn] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.stats_n_coverage:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:37:58,96] [warn] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.stats_n_coverage_primtrim:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:37:58,96] [warn] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.sc2_gene_coverage:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:37:58,97] [warn] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.consensus_qc:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:37:58,98] [warn] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.pangolin4:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:37:58,98] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-stats_n_coverage/inputs/-1676033136/ont.trimmed.rg.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/execution/ont.trimmed.rg.sorted.bam: Operation not permitted [2024-01-10 14:37:58,98] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/execution/ont.primertrimmed.rg.sorted.bam: Operation not permitted [2024-01-10 14:37:58,98] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-stats_n_coverage_primtrim/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/execution/ont.primertrimmed.rg.sorted.bam: Operation not permitted [2024-01-10 14:37:58,98] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus_qc/inputs/-1676033136/ont.medaka.consensus.fasta -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/execution/ont.medaka.consensus.fasta: Operation not permitted [2024-01-10 14:37:58,99] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.consensus_qc:NA:1]: if [ -s "" ] ; then GENOME_LEN=$(grep -v ">" | tr --delete '\n' | wc -c) elif [ ] ; then GENOME_LEN= else # set SC2 default GENOME_LEN=29903 fi # capture date and version date | tee DATE num_N=$( grep -v ">" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus_qc/inputs/-1676033136/ont.medaka.consensus.fasta | grep -o 'N' | wc -l ) if [ -z "$num_N" ] ; then num_N="0" ; fi echo $num_N | tee NUM_N num_ACTG=$( grep -v ">" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus_qc/inputs/-1676033136/ont.medaka.consensus.fasta | grep -o -E "C|A|T|G" | wc -l ) if [ -z "$num_ACTG" ] ; then num_ACTG="0" ; fi echo $num_ACTG | tee NUM_ACTG # calculate percent coverage (Wu Han-1 genome length: 29903bp) python3 -c "print ( round( ($num_ACTG / $GENOME_LEN ) * 100, 2 ) )" | tee PERCENT_REF_COVERAGE num_degenerate=$( grep -v ">" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus_qc/inputs/-1676033136/ont.medaka.consensus.fasta | grep -o -E "B|D|E|F|H|I|J|K|L|M|O|P|Q|R|S|U|V|W|X|Y|Z" | wc -l ) if [ -z "$num_degenerate" ] ; then num_degenerate="0" ; fi echo $num_degenerate | tee NUM_DEGENERATE num_total=$( grep -v ">" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus_qc/inputs/-1676033136/ont.medaka.consensus.fasta | grep -o -E '[A-Z]' | wc -l ) if [ -z "$num_total" ] ; then num_total="0" ; fi echo $num_total | tee NUM_TOTAL [2024-01-10 14:37:58,99] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.stats_n_coverage:NA:1]: date | tee DATE samtools --version | head -n1 | tee VERSION samtools stats /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-stats_n_coverage/inputs/-1676033136/ont.trimmed.rg.sorted.bam > ont.stats.txt samtools coverage /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-stats_n_coverage/inputs/-1676033136/ont.trimmed.rg.sorted.bam -m -o ont.cov.hist samtools coverage /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-stats_n_coverage/inputs/-1676033136/ont.trimmed.rg.sorted.bam -o ont.cov.txt samtools flagstat /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-stats_n_coverage/inputs/-1676033136/ont.trimmed.rg.sorted.bam > ont.flagstat.txt coverage=$(cut -f 6 ont.cov.txt | tail -n 1) depth=$(cut -f 7 ont.cov.txt | tail -n 1) meanbaseq=$(cut -f 8 ont.cov.txt | tail -n 1) meanmapq=$(cut -f 9 ont.cov.txt | tail -n 1) if [ -z "$coverage" ] ; then coverage="0" ; fi if [ -z "$depth" ] ; then depth="0" ; fi if [ -z "$meanbaseq" ] ; then meanbaseq="0" ; fi if [ -z "$meanmapq" ] ; then meanmapq="0" ; fi echo $coverage | tee COVERAGE echo $depth | tee DEPTH echo $meanbaseq | tee MEANBASEQ echo $meanmapq | tee MEANMAPQ [2024-01-10 14:37:58,99] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.stats_n_coverage_primtrim:NA:1]: date | tee DATE samtools --version | head -n1 | tee VERSION samtools stats /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-stats_n_coverage_primtrim/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam > ont.stats.txt samtools coverage /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-stats_n_coverage_primtrim/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam -m -o ont.cov.hist samtools coverage /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-stats_n_coverage_primtrim/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam -o ont.cov.txt samtools flagstat /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-stats_n_coverage_primtrim/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam > ont.flagstat.txt coverage=$(cut -f 6 ont.cov.txt | tail -n 1) depth=$(cut -f 7 ont.cov.txt | tail -n 1) meanbaseq=$(cut -f 8 ont.cov.txt | tail -n 1) meanmapq=$(cut -f 9 ont.cov.txt | tail -n 1) if [ -z "$coverage" ] ; then coverage="0" ; fi if [ -z "$depth" ] ; then depth="0" ; fi if [ -z "$meanbaseq" ] ; then meanbaseq="0" ; fi if [ -z "$meanmapq" ] ; then meanmapq="0" ; fi echo $coverage | tee COVERAGE echo $depth | tee DEPTH echo $meanbaseq | tee MEANBASEQ echo $meanmapq | tee MEANMAPQ [2024-01-10 14:37:58,99] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-pangolin4/inputs/-1676033136/ont.medaka.consensus.fasta -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/execution/ont.medaka.consensus.fasta: Operation not permitted [2024-01-10 14:37:59,00] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.sc2_gene_coverage:NA:1]: samtools index /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam chr=$(samtools idxstats /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam | cut -f 1 | head -1) samtools coverage -r "${chr}:21563-25384" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam >> ont.cov.txt s_gene_depth=$(cut -f 7 ont.cov.txt | tail -n 1) # samtools outputs 3 columns; column 3 is the depth of coverage per nucleotide position, piped to awk to count the positions # above min_depth, then wc -l counts them all orf1ab=$(samtools depth -J -r "${chr}:266-21555" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) sgene=$(samtools depth -J -r "${chr}:21563-25384" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) orf3a=$(samtools depth -J -r "${chr}:25393-26220" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) egene=$(samtools depth -J -r "${chr}:26245-26472" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) mgene=$(samtools depth -J -r "${chr}:26523-27191" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) orf6=$(samtools depth -J -r "${chr}:27202-27387" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) orf7a=$(samtools depth -J -r "${chr}:27394-27759" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) orf7b=$(samtools depth -J -r "${chr}:27756-27887" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) orf8=$(samtools depth -J -r "${chr}:27894-28259" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) ngene=$(samtools depth -J -r "${chr}:28274-29533" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) orf10=$(samtools depth -J -r "${chr}:29558-29674" /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-sc2_gene_coverage/inputs/-1676033136/ont.primertrimmed.rg.sorted.bam | awk -F "\t" '{if ($3 > 20) print;}' | wc -l ) orf1ab_pc=$(python3 -c "print ( round( ($orf1ab / 21290 ) * 100, 2 ) )") sgene_pc=$(python3 -c "print ( round( ($sgene / 3822 ) * 100, 2 ) )") orf3a_pc=$(python3 -c "print ( round( ($orf3a / 828 ) * 100, 2 ) )") egene_pc=$(python3 -c "print ( round( ($egene / 228 ) * 100, 2 ) )") mgene_pc=$(python3 -c "print ( round( ($mgene / 669 ) * 100, 2 ) )") orf6_pc=$(python3 -c "print ( round( ($orf6 / 186 ) * 100, 2 ) )") orf7a_pc=$(python3 -c "print ( round( ($orf7a / 366 ) * 100, 2 ) )") orf7b_pc=$(python3 -c "print ( round( ($orf7b / 132 ) * 100, 2 ) )") orf8_pc=$(python3 -c "print ( round( ($orf8 / 366 ) * 100, 2 ) )") ngene_pc=$(python3 -c "print ( round( ($ngene / 1260 ) * 100, 2 ) )") orf10_pc=$(python3 -c "print ( round( ($orf10 / 117 ) * 100, 2 ) )") echo -e "#NOTE: THE VALUES BELOW ASSUME WUHAN-1 REFERENCE GENOME" > ont.percent_gene_coverage.tsv echo -e "Gene\tPercent_Coverage" >> ont.percent_gene_coverage.tsv echo -e "ORF1ab\t" $orf1ab_pc >> ont.percent_gene_coverage.tsv echo -e "S_gene\t" $sgene_pc >> ont.percent_gene_coverage.tsv echo -e "ORF3a\t" $orf3a_pc >> ont.percent_gene_coverage.tsv echo -e "E_gene\t" $egene_pc >> ont.percent_gene_coverage.tsv echo -e "M_gene\t" $mgene_pc >> ont.percent_gene_coverage.tsv echo -e "ORF6\t" $orf6_pc >> ont.percent_gene_coverage.tsv echo -e "ORF7a\t" $orf7a_pc >> ont.percent_gene_coverage.tsv echo -e "ORF7b\t" $orf7b_pc >> ont.percent_gene_coverage.tsv echo -e "ORF8\t" $orf8_pc >> ont.percent_gene_coverage.tsv echo -e "N_gene\t" $ngene_pc >> ont.percent_gene_coverage.tsv echo -e "ORF10\t" $orf10_pc >> ont.percent_gene_coverage.tsv if [ -z "s_gene_depth" ] ; then s_gene_depth="0"; fi echo $s_gene_depth | tee S_GENE_DEPTH echo $sgene_pc | tee S_GENE_PC [2024-01-10 14:37:59,01] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.pangolin4:NA:1]: set -e # date and version capture date | tee DATE { pangolin --all-versions && usher --version; } | tr '\n' ';' | cut -f -6 -d ';' | tee VERSION_PANGOLIN_ALL pangolin "/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-pangolin4/inputs/-1676033136/ont.medaka.consensus.fasta" \ \ --min-length 10000 \ --max-ambig 0.5 \ --expanded-lineage \ \ \ --outfile "ont.pangolin_report.csv" \ --verbose \ python3 < /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/execution/ont.medaka.consensus.fasta: Operation not permitted [2024-01-10 14:37:59,09] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.nextclade:NA:1]: NEXTCLADE_VERSION="$(nextclade --version)" echo $NEXTCLADE_VERSION > NEXTCLADE_VERSION nextclade dataset get --name="sars-cov-2" --reference="MN908947" --tag="2023-09-21T12:00:00Z" -o nextclade_dataset_dir --verbose set -e nextclade run \ --input-dataset=nextclade_dataset_dir/ \ \ \ \ \ \ \ --output-json "ont.medaka.consensus".nextclade.json \ --output-tsv "ont.medaka.consensus".nextclade.tsv \ --output-tree "ont.medaka.consensus".nextclade.auspice.json \ --output-all=. \ "/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nextclade/inputs/-1676033136/ont.medaka.consensus.fasta" [2024-01-10 14:37:59,12] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.nextclade:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nextclade/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nextclade/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nextclade:/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nextclade:delegated \ nextstrain/nextclade@sha256:f4440021c7f854433bd56433024f38199de27e0ef657e11d5fb28aad4f265a48 /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nextclade/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nextclade/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nextclade/execution/docker_cid # return exit code exit $rc [2024-01-10 14:38:03,52] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.consensus_qc:NA:1]: job id: 65354 [2024-01-10 14:38:03,52] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.stats_n_coverage_primtrim:NA:1]: job id: 65360 [2024-01-10 14:38:03,52] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.pangolin4:NA:1]: job id: 65388 [2024-01-10 14:38:03,52] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.nextclade:NA:1]: job id: 65407 [2024-01-10 14:38:03,52] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.sc2_gene_coverage:NA:1]: job id: 65349 [2024-01-10 14:38:03,52] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.stats_n_coverage:NA:1]: job id: 65380 [2024-01-10 14:38:03,52] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.consensus_qc:NA:1]: Status change from - to Done [2024-01-10 14:38:03,52] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.nextclade:NA:1]: Status change from - to Done [2024-01-10 14:38:03,52] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.stats_n_coverage_primtrim:NA:1]: Status change from - to Done [2024-01-10 14:38:03,52] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.pangolin4:NA:1]: Status change from - to WaitingForReturnCode [2024-01-10 14:38:03,52] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.stats_n_coverage:NA:1]: Status change from - to Done [2024-01-10 14:38:03,53] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.sc2_gene_coverage:NA:1]: Status change from - to Done [2024-01-10 14:38:05,29] [info] WorkflowExecutionActor-a74105f8-2597-4528-bb3c-9eb852973d92 [a74105f8]: Starting theiacov_ont.nanoplot_raw, theiacov_ont.vadr, theiacov_ont.nextclade_output_parser, theiacov_ont.nanoplot_clean [2024-01-10 14:38:08,66] [info] Assigned new job execution tokens to the following groups: a74105f8: 4 [2024-01-10 14:38:08,91] [warn] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.vadr:NA:1]: Unrecognized runtime attribute keys: dx_instance_type, cpu, memory [2024-01-10 14:38:08,91] [warn] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.nanoplot_raw:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:38:08,91] [warn] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.nanoplot_clean:NA:1]: Unrecognized runtime attribute keys: preemptible, disk, disks, cpu, memory [2024-01-10 14:38:08,91] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-vadr/inputs/-1676033136/ont.medaka.consensus.fasta -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-consensus/execution/ont.medaka.consensus.fasta: Operation not permitted [2024-01-10 14:38:08,92] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nanoplot_clean/inputs/1986200811/artic_ncov2019_ont.fastq -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-read_qc_trim/read_QC_trim_ont/dbaaa31c-b302-4d0d-9600-5d4ddf59c076/call-read_filtering/execution/artic_ncov2019_ont.fastq: Operation not permitted [2024-01-10 14:38:08,92] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.nanoplot_raw:NA:1]: # get version NanoPlot --version | tee "VERSION" # run nanoplot # --prefix for output file tag # --threads for number of threads allowed # --N50 to display N50 mark in read length histogram # --loglength to show logarithmic scaling of lengths # --tsv_stats to output the stats file in TSV format # --maxlength to hide reads longer than this NanoPlot \ --fastq /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nanoplot_raw/inputs/1032112200/ont.fastq.gz \ --prefix "ont_" \ --threads 4 \ --N50 \ --loglength \ --tsv_stats \ --maxlength 100000 # grep read statistics from tsv stats file grep "number_of_reads" ont_NanoStats.txt | cut -f 2 | tee NUMBER_OF_READS NUM_BASES=$(grep "number_of_bases" ont_NanoStats.txt | cut -f 2 | tee NUMBER_OF_BASES) grep "mean_read_length" ont_NanoStats.txt | cut -f 2 | tee MEAN_READ_LENGTH grep "mean_qual" ont_NanoStats.txt | cut -f 2 | tee MEAN_QUAL # estimate coverage # using math: C = N / G where N is number of bases, and G is estimated genome size python3 -c "print(round(${NUM_BASES} / 29903, 2))" | tee EST_COVERAGE [2024-01-10 14:38:08,92] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.vadr:NA:1]: set -e if [ 27125 -gt 10000 ]; then # remove terminal ambiguous nucleotides /opt/vadr/vadr/miniscripts/fasta-trim-terminal-ambigs.pl \ "/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-vadr/inputs/-1676033136/ont.medaka.consensus.fasta" \ --minlen 50 \ --maxlen 30000 \ > "ont.medaka.consensus_trimmed.fasta" # run VADR # --split and --cpu must be used in conjuction v-annotate.pl \ --split --cpu 2 \ --noseqnamemax --glsearch -s -r --nomisc --mkey sarscov2 --lowsim5seq 6 --lowsim3seq 6 --alt_fail lowscore,insertnn,deletinn --out_allfasta \ "ont.medaka.consensus_trimmed.fasta" \ "ont.medaka.consensus" # package everything for output tar -C "ont.medaka.consensus" -czvf "ont.medaka.consensus.vadr.tar.gz" . # package up FASTA files into zip file for output. Note: this will work whether the --out_allfasta flag is included or not (there are just more when the option is used) mkdir -v vadr_fasta_files cp -v ont.medaka.consensus/*.fa vadr_fasta_files zip ont.medaka.consensus_vadr-fasta-files.zip vadr_fasta_files/*.fa # prep alerts into a tsv file for parsing cut -f 5 "ont.medaka.consensus/ont.medaka.consensus.vadr.alt.list" | tail -n +2 > "ont.medaka.consensus.vadr.alerts.tsv" cat "ont.medaka.consensus.vadr.alerts.tsv" | wc -l > NUM_ALERTS else echo "VADR skipped due to poor assembly; assembly length (unambiguous) = 27125" > NUM_ALERTS fi [2024-01-10 14:38:08,93] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.vadr:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-vadr/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-vadr/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-vadr:/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-vadr:delegated \ us-docker.pkg.dev/general-theiagen/staphb/vadr@sha256:f15a73a6f6e0802c3a9c05e7c602740e7029a2e1ccc14aa6935b459be87c98d7 /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-vadr/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-vadr/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-vadr/execution/docker_cid # return exit code exit $rc [2024-01-10 14:38:08,95] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.nanoplot_raw:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nanoplot_raw/execution/docker_cid # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nanoplot_raw/execution/docker_cid \ -i \ \ --entrypoint /bin/bash \ -v /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nanoplot_raw:/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nanoplot_raw:delegated \ us-docker.pkg.dev/general-theiagen/staphb/nanoplot@sha256:68edf4b3e515b179fa12ec8405ba196af204625e2f30b7e5512427f773c81d79 /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nanoplot_raw/execution/script # get the return code (working even if the container was detached) rc=$(docker wait cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nanoplot_raw/execution/docker_cid) # remove the container after waiting docker rm cat /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nanoplot_raw/execution/docker_cid # return exit code exit $rc [2024-01-10 14:38:08,96] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.nanoplot_clean:NA:1]: # get version NanoPlot --version | tee "VERSION" # run nanoplot # --prefix for output file tag # --threads for number of threads allowed # --N50 to display N50 mark in read length histogram # --loglength to show logarithmic scaling of lengths # --tsv_stats to output the stats file in TSV format # --maxlength to hide reads longer than this NanoPlot \ --fastq /cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nanoplot_clean/inputs/1986200811/artic_ncov2019_ont.fastq \ --prefix "ont_" \ --threads 4 \ --N50 \ --loglength \ --tsv_stats \ --maxlength 100000 # grep read statistics from tsv stats file grep "number_of_reads" ont_NanoStats.txt | cut -f 2 | tee NUMBER_OF_READS NUM_BASES=$(grep "number_of_bases" ont_NanoStats.txt | cut -f 2 | tee NUMBER_OF_BASES) grep "mean_read_length" ont_NanoStats.txt | cut -f 2 | tee MEAN_READ_LENGTH grep "mean_qual" ont_NanoStats.txt | cut -f 2 | tee MEAN_QUAL # estimate coverage # using math: C = N / G where N is number of bases, and G is estimated genome size python3 -c "print(round(${NUM_BASES} / 29903, 2))" | tee EST_COVERAGE [2024-01-10 14:38:08,97] [warn] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.nextclade_output_parser:NA:1]: Unrecognized runtime attribute keys: disk, dx_instance_type, disks, cpu, memory [2024-01-10 14:38:08,98] [warn] Localization via hard link has failed: /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nextclade_output_parser/inputs/-386462497/ont.medaka.consensus.nextclade.tsv -> /home/curtis_kapsak/github/public_health_bioinformatics/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nextclade/execution/ont.medaka.consensus.nextclade.tsv: Operation not permitted [2024-01-10 14:38:08,98] [info] BackgroundConfigAsyncJobExecutionActor [a74105f8theiacov_ont.nextclade_output_parser:NA:1]: # Set WDL input variable to input.tsv file cat "/cromwell-executions/theiacov_ont/a74105f8-2597-4528-bb3c-9eb852973d92/call-nextclade_output_parser/inputs/-386462497/ont.medaka.consensus.nextclade.tsv" > input.tsv touch TAMIFLU_AASUBS # Parse outputs using python3 python3 <

✅ theiacov_ont via cromwell was successful and exit code was 0.

That wraps up all the cromwell tests that failed via the GHActions runner VMs, presumably due to disk space limitations, but passed on my local CI environment.

We are good to merge 🎉

  • © Githubissues.
  • Githubissues is a development platform for aggregating issues.