bio-boris commented 3 years ago

Description of PR purpose/changes

prepping for batch running throughout the code base, starting with the preflight checks
removing support for wsid in batch jobs outside of batchjobparams
this pr in favor of #399
the first pass is to make running jobs preflight checks run from a list of jobs.
Then the second pass is to make the jobs all return a series of docs to save into mongo with a bulk operation.
Then the next pass is to make them all get submitted to condor with a bulk operation.
Then the next pass is to make this operation done via a thread.
Then the next pass is to make sure the retry_multiple is submitting all jobs via this bulk submit.
Then the next pass is to make sure that batch submits via the bulk submit.

Jira Ticket / Github Issue

[ ] Added the Jira Ticket to the title of the PR e.g. (DATAUP-69 Adds a PR template)

Testing Instructions

Details for how to test the PR:
[ ] Tests pass in Github Actions and locally
[ ] Changes available by spinning up a local test suite and doing X

Dev Checklist:

[x] My code follows the guidelines at https://sites.google.com/truss.works/kbasetruss/data-upload-project/development
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] Any dependent changes have been merged and published in downstream modules
[x] I have run Black and Flake8 on changed Python Code manually or with git precommit (and the Github Actions build passes)

Updating Version and Release Notes (if applicable)

[ ] Version has been bumped for each release
[ ] Release notes have been updated for each release (and during the merge of feature branches)

lgtm-com[bot] commented 3 years ago

This pull request introduces 3 alerts when merging 7959e77144edbc969f857557ae5785d81e2fdeae into bd218d67be688a27152bf99e6538cb98ee6654e0 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Unused import
1 for Unreachable code

lgtm-com[bot] commented 3 years ago

This pull request introduces 2 alerts when merging 3e8c405738fb3a1e3b8f28f3616837d1f7391fc6 into bd218d67be688a27152bf99e6538cb98ee6654e0 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Unreachable code

lgtm-com[bot] commented 3 years ago

This pull request introduces 2 alerts when merging ffc65802dadead3708d6c030cdb82da0ceeef148 into bd218d67be688a27152bf99e6538cb98ee6654e0 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Unreachable code

codecov[bot] commented 3 years ago

Codecov Report

Merging #402 (44f7e97) into develop (1307bce) will increase coverage by 4.39%. The diff coverage is 93.13%.

:exclamation: Current head 44f7e97 differs from pull request most recent head f334e98. Consider uploading reports for the commit f334e98 to get more accurate results

@@             Coverage Diff             @@
##           develop     #402      +/-   ##
===========================================
+ Coverage    83.84%   88.23%   +4.39%     
===========================================
  Files           22       29       +7     
  Lines         1993     2712     +719     
===========================================
+ Hits          1671     2393     +722     
+ Misses         322      319       -3

Impacted Files	Coverage Δ
lib/execution_engine2/utils/CondorTuples.py	`100.00% <ø> (ø)`
lib/execution_engine2/utils/SlackUtils.py	`35.71% <8.33%> (-7.53%)`	:arrow_down:
lib/execution_engine2/authclient.py	`74.57% <74.57%> (ø)`
lib/execution_engine2/sdk/EE2Status.py	`86.19% <76.31%> (-2.16%)`	:arrow_down:
lib/execution_engine2/db/models/models.py	`92.51% <81.25%> (-1.21%)`	:arrow_down:
lib/execution_engine2/db/MongoUtil.py	`71.82% <87.09%> (+4.74%)`	:arrow_up:
lib/execution_engine2/sdk/EE2Runjob.py	`93.29% <92.23%> (+8.48%)`	:arrow_up:
lib/execution_engine2/sdk/EE2Logs.py	`96.55% <93.75%> (+5.44%)`	:arrow_up:
lib/execution_engine2/sdk/SDKMethodRunner.py	`83.96% <95.29%> (+4.63%)`	:arrow_up:
lib/execution_engine2/utils/Condor.py	`87.79% <96.82%> (-1.10%)`	:arrow_down:
... and 32 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 8d4299f...f334e98. Read the comment docs.

lgtm-com[bot] commented 3 years ago

This pull request introduces 2 alerts when merging fbb846ac5b5806b129871c99b1da26717914c2a5 into bd218d67be688a27152bf99e6538cb98ee6654e0 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Unreachable code