search

seqeralabs / seqera-kit

Python implementation and field-tool for automated pipeline launching through Tower CLI (beta)
Apache License 2.0
25 stars 6 forks source link

feat(SeqeraPlatform): seqerakit can write JSON to stdout for detailed records #146

Closed adamrtalbot closed 2 weeks ago

adamrtalbot commented 6 months ago

We needed a method of storing run info from the seqerakit CLI. After this change, you can use the '--json' flag to write the JSON return from the CLI to STDOUT. This can be captured and used in a different process or piped. This is enabled by a new command line flag, --json but should be possible when instantiating the SeqeraPlatform class in Python. The to_json kwarg is removed and replaced with this method so this might break some uses. The JSON is an attribute of the class and can be referenced whenever you use the SeqeraPlatform class.

BREAKING CHANGE: This could break anyone using the Python implementation with to_json=True. This is mainly me.

To use, try running seqerakit with the flag -j and piping:

seqerakit -j launch/*.yml | jq --slurp > launched-pipelines.json

(update): Because the function is overlapping with to_json, we just use either will create JSON output. If to_json is used it will print to stderr, if -j is used it will write to stdout suitable for piping.

adamrtalbot commented 6 months ago

Appears to double up the -o json part sometimes:

ERROR:root:Resource creation failed: 'option '--output' (<output>) should be specified only once`
ejseqera commented 3 months ago

@adamrtalbot I like this implementation, it might break things but it's a feature worth adding. What's left to do?

adamrtalbot commented 3 months ago

@adamrtalbot I like this implementation, it might break things but it's a feature worth adding. What's left to do?

Fix the doubling of --output json twice is the main issue.

Ideally, we might cache the output and write a valid JSON as a list although piping to jq seems acceptable for a first draft.

adamrtalbot commented 2 months ago

Trying to finish this off and I'm confused about how I should handle the to_json flag in sp._execute_command(): https://github.com/seqeralabs/seqera-kit/blob/0271c38efe18e62d3286b510e2766cb4c5b88d6c/seqerakit/seqeraplatform.py#L103-L127

It seems like these have overlapping purposes so I've added an or statement which fixes the tests: https://github.com/seqeralabs/seqera-kit/pull/146/commits/1131da8c83584e479061c59335273e3ce650a585, but not sure if this is the right thing to do.

adamrtalbot commented 2 weeks ago

Yo @ejseqera this one is good to go and I think it would be useful with our current projects. Let me know what else it needs.

adamrtalbot commented 1 week ago

Ah heck, this still doubles up if you use overwrite: true:

INFO:root: Running command: tw -o json -o json pipelines list -w $ORGANIZATION_NAME/$WORKSPACE_NAME
ERROR:root:Resource creation failed: 'option '--output' (<output>) should be specified only once