yuch7 / cwlexec

A new open source tool to run CWL workflows on LSF
Other
36 stars 8 forks source link

Cannot support to run cwlexec command concurrently by one user #18

Closed skeeey closed 6 years ago

skeeey commented 6 years ago

The cwlexec command cannot be executed concurrently by one user, this because cwlexec use HyperSQL (file: database) to record the workflow execution information. (the db file is in the $HOME/.cwlexec by default) , but HyperSQL file model doesn't support to write db concurrently. more information can be found from: http://hsqldb.org/doc/2.0/guide/running-chapt.html#N100CF

skeeey commented 6 years ago

Note: this solution is obsolete

Thanks @drjrm3's suggestion

Because HyperSQL file model cannot support to write a database concurrently, we add an option (-db) for cwlexec (see 808061e), with this option, users can define the HyperSQL database file directory by themselves.

There is an example:

Define a CWLCommmandLineTool (named concurrent.cwl)

cwlVersion: v1.0
class: CommandLineTool

baseCommand: ["sleep", "30"]

inputs: []
outputs: []

We use cwlexec to run it parallelly by one user. we opened two terminals and run it by

cwlexec concurrent.cwl

it can run successfully on one terminal, and the other reports

Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment]

Then we use -db option, on one terminal, we run it by

cwlexec concurrent.cwl

and on the other terminal, we create a directory (e.g. mkdir -p /tmp/cwlexec/weliu/concurrent0) firstly, then we run it by

cwlexec -db /tmp/cwlexec/weliu/concurrent0 concurrent.cwl

Both of them can be executed successfully

Note: By this way, the workflow execution records are stored in different directory, so if you want to use --list or --rerun, you need to use -db option to specify where the workflow execution records are stored, e.g. list the executed workflows

cwlexec -l -db /tmp/cwlexec/weliu/concurrent0
skeeey commented 6 years ago

The above way separates the flow execution records for one user, so we decide to use H2 database instead of HyperSQL

The H2 database with Automatic Mixed Mode can resolve our problem, see 5a5ff60 and 231960a