search

vincentclaes / datajob

Build and deploy a serverless data pipeline on AWS with no effort.
https://pypi.org/project/datajob/
Apache License 2.0
110 stars 19 forks source link

94 execute sagemaker #105

Closed vincentclaes closed 3 years ago

sourcery-ai[bot] commented 3 years ago

Sourcery Code Quality Report

❌  Merging this PR will decrease code quality in the affected files by 0.01%.

Quality metrics Before After Change
Complexity 0.84 ⭐ 0.96 ⭐ 0.12 👎
Method Length 41.29 ⭐ 41.80 ⭐ 0.51 👎
Working memory 7.72 🙂 7.67 🙂 -0.05 👍
Quality 78.16% ⭐ 78.15% ⭐ -0.01% 👎
Other metrics Before After Change
Lines 2372 2634 262
Changed files Quality Before Quality After Quality Change
datajob/init.py 82.38% ⭐ 82.38% ⭐ 0.00%
datajob/datajob.py 76.57% ⭐ 76.27% ⭐ -0.30% 👎
datajob/datajob_base.py 85.90% ⭐ 86.93% ⭐ 1.03% 👍
datajob/datajob_context.py 78.90% ⭐ 78.90% ⭐ 0.00%
datajob/datajob_stack.py 88.12% ⭐ 87.94% ⭐ -0.18% 👎
datajob/glue/glue_job.py 74.77% 🙂 74.83% 🙂 0.06% 👍
datajob/package/wheel.py 89.29% ⭐ 89.29% ⭐ 0.00%
datajob/sagemaker/init.py 82.10% ⭐ 82.14% ⭐ 0.04% 👍
datajob/sagemaker/sagemaker_job.py 81.34% ⭐ 80.86% ⭐ -0.48% 👎
datajob/sns/sns.py 88.08% ⭐ 88.08% ⭐ 0.00%
datajob/stepfunctions/stepfunctions_execute.py 86.02% ⭐ 81.57% ⭐ -4.45% 👎
datajob/stepfunctions/stepfunctions_workflow.py 82.46% ⭐ 82.47% ⭐ 0.01% 👍
datajob_tests/test_datajob_context.py 79.89% ⭐ 79.89% ⭐ 0.00%
datajob_tests/test_datajob_stack.py 89.90% ⭐ 89.90% ⭐ 0.00%
datajob_tests/datajob_cli_tests/test_datajob_deploy.py 94.56% ⭐ 94.56% ⭐ 0.00%
datajob_tests/datajob_cli_tests/test_datajob_execute.py 81.47% ⭐ 77.00% ⭐ -4.47% 👎
datajob_tests/glue/test_glue_job.py 89.95% ⭐ 89.95% ⭐ 0.00%
datajob_tests/sagemaker/test_sagemaker.py 76.56% ⭐ 73.24% 🙂 -3.32% 👎
datajob_tests/sagemaker/resources/processing.py 59.89% 🙂 59.89% 🙂 0.00%
datajob_tests/sagemaker/resources/train.py 71.32% 🙂 71.32% 🙂 0.00%
datajob_tests/stepfunctions/test_stepfunctions.py 67.25% 🙂 67.25% 🙂 0.00%
examples/data_pipeline_parallel/datajob_stack.py 68.73% 🙂 68.73% 🙂 0.00%
examples/data_pipeline_parallel/glue_jobs/task.py 99.42% ⭐ 99.42% ⭐ 0.00%
examples/data_pipeline_pyspark/datajob_stack.py 62.67% 🙂 62.67% 🙂 0.00%
examples/data_pipeline_pyspark/setup.py 82.30% ⭐ 82.30% ⭐ 0.00%
examples/data_pipeline_pyspark/glue_job/glue_pyspark_example.py 85.10% ⭐ 85.10% ⭐ 0.00%
examples/data_pipeline_simple/datajob_stack.py 81.52% ⭐ 81.52% ⭐ 0.00%
examples/data_pipeline_simple/datajob_stack_explicit.py 80.28% ⭐ 80.28% ⭐ 0.00%
examples/data_pipeline_simple/glue_jobs/task1.py 99.42% ⭐ 99.42% ⭐ 0.00%
examples/data_pipeline_simple/glue_jobs/task2.py 99.42% ⭐ 99.42% ⭐ 0.00%
examples/data_pipeline_simple/glue_jobs/task3.py 99.42% ⭐ 99.42% ⭐ 0.00%
examples/data_pipeline_with_packaged_project/datajob_stack.py 77.17% ⭐ 77.17% ⭐ 0.00%
examples/data_pipeline_with_packaged_project/datajob_stack_explicit.py 73.03% 🙂 73.03% 🙂 0.00%
examples/data_pipeline_with_packaged_project/setup.py 82.86% ⭐ 82.86% ⭐ 0.00%
examples/data_pipeline_with_packaged_project/glue_jobs/glue_helper.py 100.00% ⭐ 100.00% ⭐ 0.00%
examples/data_pipeline_with_packaged_project/glue_jobs/task1.py 99.20% ⭐ 99.20% ⭐ 0.00%
examples/data_pipeline_with_packaged_project/glue_jobs/task2.py 99.20% ⭐ 99.20% ⭐ 0.00%
examples/ml_pipeline_sagemaker_scikitlearn/datajob_stack.py 44.58% 😞 44.58% 😞 0.00%
examples/ml_pipeline_sagemaker_scikitlearn/resources/preprocessing.py 54.80% 🙂 54.80% 🙂 0.00%
examples/ml_pipeline_sagemaker_scikitlearn/resources/train.py 71.32% 🙂 71.32% 🙂 0.00%

Here are some functions in these files that still need a tune-up:

File Function Complexity Length Working Memory Quality Recommendation
datajob_tests/sagemaker/resources/processing.py main 4 ⭐ 205 ⛔ 10 😞 54.06% 🙂 Try splitting into smaller methods. Extract out complex expressions
datajob/glue/glue_job.py GlueJob.__init__ 1 ⭐ 122 😞 18 ⛔ 54.28% 🙂 Try splitting into smaller methods. Extract out complex expressions
datajob_tests/stepfunctions/test_stepfunctions.py TestStepfunctions.test_create_tasks_for_orchestration_simple_flow_successfully 0 ⭐ 156 😞 14 😞 55.68% 🙂 Try splitting into smaller methods. Extract out complex expressions
datajob/sagemaker/sagemaker_job.py ProcessingStep.__init__ 1 ⭐ 81 🙂 19 ⛔ 59.09% 🙂 Extract out complex expressions
datajob/glue/glue_job.py GlueJob._create_glue_job 1 ⭐ 88 🙂 17 ⛔ 59.70% 🙂 Extract out complex expressions

Legend and Explanation

The emojis denote the absolute quality of the code:

The 👍 and 👎 indicate whether the quality has improved or gotten worse with this pull request.

Please see our documentation here for details on how these metrics are calculated.

We are actively working on this report - lots more documentation and extra metrics to come!

Help us improve this quality report!