aws / sagemaker-experiments

Experiment tracking and metric logging for Amazon SageMaker notebooks and model training.
Apache License 2.0
125 stars 36 forks source link

add collect tests tox env #115

Closed danabens closed 3 years ago

danabens commented 3 years ago

collect-tests is used for some IDE test integrations

--- aws/sagemaker-experiments ‹main* M› » tox -e collect-tests                                                                                                                                        1 ↵
GLOB sdist-make: /Users/danabens/workplace/github/aws/sagemaker-experiments/setup.py
collect-tests create: /Users/danabens/workplace/github/aws/sagemaker-experiments/.tox/collect-tests
collect-tests installdeps: boto3 >= 1.12.8, python-dateutil, pytest, pytest-cov, docker
collect-tests inst: /Users/danabens/workplace/github/aws/sagemaker-experiments/.tox/.tmp/package/1/sagemaker-experiments-0.1.26.dev7+g9363166.d20210122.zip
collect-tests installed: apipkg==1.5,appdirs==1.4.4,atomicwrites==1.4.0,attrs==20.3.0,boto3==1.16.59,botocore==1.19.59,certifi==2020.12.5,chardet==4.0.0,coverage==5.3.1,distlib==0.3.1,docker==4.4.1,execnet==1.7.1,filelock==3.0.12,flake8==3.8.4,idna==2.10,importlib-metadata==0.23,iniconfig==1.1.1,jmespath==0.10.0,joblib==1.0.0,mccabe==0.6.1,more-itertools==8.6.0,numpy==1.19.5,packaging==20.8,pandas==1.2.1,pluggy==0.13.1,py==1.10.0,pycodestyle==2.6.0,pyflakes==2.2.0,pyparsing==2.4.7,pytest==4.4.1,pytest-cov==2.9.0,pytest-cover==3.0.0,pytest-coverage==0.0,pytest-forked==1.3.0,pytest-rerunfailures==8.0,pytest-xdist==1.34.0,python-dateutil==2.8.1,pytz==2020.5,requests==2.25.1,s3transfer==0.3.4,sagemaker-experiments @ file:///Users/danabens/workplace/github/aws/sagemaker-experiments/.tox/.tmp/package/1/sagemaker-experiments-0.1.26.dev7%2Bg9363166.d20210122.zip,scikit-learn==0.24.1,scipy==1.6.0,six==1.15.0,sklearn==0.0,threadpoolctl==2.1.0,toml==0.10.2,tox==3.13.1,urllib3==1.26.2,virtualenv==20.4.0,websocket-client==0.57.0,zipp==3.4.0
collect-tests run-test-pre: PYTHONHASHSEED='608801620'
collect-tests run-test: commands[0] | pytest --collect-only
========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.8.0, pytest-4.4.1, py-1.10.0, pluggy-0.13.1
cachedir: .tox/collect-tests/.pytest_cache
rootdir: /Users/danabens/workplace/github/aws/sagemaker-experiments, inifile: pytest.ini
plugins: cov-2.9.0, rerunfailures-8.0, xdist-1.34.0, forked-1.3.0
collected 181 items
<Package /Users/danabens/workplace/github/aws/sagemaker-experiments/tests/integ>
  <Module test_experiment.py>
    <Function test_create_delete>
    <Function test_create_tags>
    <Function test_save>
    <Function test_save_load>
    <Function test_list>
    <Function test_list_sort>
    <Function test_search>
    <Function test_create_trial>
    <Function test_list_trials>
    <Function test_delete_all>
    <Function test_delete_all_fails>
  <Module test_search_expression.py>
    <Function test_search>
    <Function test_nested_search>
    <Function test_sub_expression>
  <Module test_track_from_processing_job.py>
    <Function test_track_from_processing_job>
  <Module test_track_from_training_job.py>
    <Function test_track_from_training_job>
  <Module test_tracker.py>
    <Function test_load_trial_component>
    <Function test_load_training_job>
    <Function test_load_trial_component_fails>
    <Function test_create>
    <Function test_create_display_name>
    <Function test_log_simple>
    <Function test_log_artifact>
    <Function test_log_artifacts>
    <Function test_create_default_bucket>
    <Function test_create_lineage_artifacts>
    <Function test_log_table_artifact>
  <Module test_training_job.py>
    <Function test_search>
  <Module test_trial.py>
    <Function test_create_delete>
    <Function test_create_tags>
    <Function test_list>
    <Function test_list_with_trial_component>
    <Function test_list_sort>
    <Function test_search>
    <Function test_add_remove_trial_component>
    <Function test_save>
  <Module test_trial_component.py>
    <Function test_create_delete>
    <Function test_create_tags>
    <Function test_delete_with_force_disassociate>
    <Function test_save>
    <Function test_load>
    <Function test_list_sort>
    <Function test_list_empty_trials>
    <Function test_list_trials>
    <Function test_list_trial_components_by_experiment>
    <Function test_search>
<Package /Users/danabens/workplace/github/aws/sagemaker-experiments/tests/unit>
  <Module test_api_types.py>
    <Function test_parameter_str_string>
    <Function test_parameter_str_number>
    <Function test_parameter_str_none>
  <Module test_base_types.py>
    <Function test_from_boto>
    <Function test_to_boto>
    <Function test_custom_type>
    <Function test_custom_type_list>
    <Function test_custom_type_dict>
    <Function test_construct>
    <Function test_update>
    <Function test_delete>
    <Function test_list_empty>
    <Function test_list_with_items>
    <Function test_list_with_next_token>
    <Function test_list_no_client>
  <Module test_environment.py>
    <Function test_processing_job_environment>
    <Function test_training_job_environment>
    <Function test_no_environment>
    <Function test_resolve_trial_component>
    <Function test_resolve_trial_component_fails>
  <Module test_experiment.py>
    <Function test_load>
    <Function test_create>
    <Function test_create_with_tags>
    <Function test_list>
    <Function test_list_trials_empty>
    <Function test_list_trials_single>
    <Function test_list_trials_two_values>
    <Function test_next_token>
    <Function test_list_trials_call_args>
    <Function test_search>
    <Function test_experiment_create_trial_with_name>
    <Function test_experiment_create_trial_with_prefix>
    <Function test_save>
    <Function test_delete>
    <Function test_delete_all_with_incorrect_action_name>
    <Function test_delete_all>
    <Function test_delete_all_fail>
  <Module test_list_trial_components_from_trial.py>
    <Function test_list_trial_components>
    <Function test_list_trial_components_empty>
    <Function test_list_trial_components_single>
    <Function test_list_trial_components_two_values>
    <Function test_next_token>
    <Function test_list_trial_components_call_args>
  <Module test_metrics.py>
    <Function test_RawMetricData_utc_timestamp>
    <Function test_RawMetricData_aware_timestamp>
    <Function test_RawMetricData_naive_timestamp>
    <Function test_RawMetricData_number_timestamp>
    <Function test_RawMetricData_invalid_timestamp>
    <Function test_file_metrics_writer_log_metric>
    <Function test_file_metrics_writer_flushes_buffer_every_line_log_metric>
    <Function test_file_metrics_writer_context_manager>
    <Function test_file_metrics_writer_fail_write_on_close>
    <Function test_file_metrics_writer_no_write>
  <Module test_search_expression.py>
    <Function test_filters>
    <Function test_partial_filters>
    <Function test_nested_filters>
    <Function test_search_expression>
    <Function test_illegal_search_expression>
  <Module test_tracker.py>
    <Function test_load_in_sagemaker_training_job>
    <Function test_load_in_sagemaker_processing_job>
    <Function test_load_in_sagemaker_job_no_resolved_tc>
    <Function test_load>
    <Function test_load_by_training_job_name>
    <Function test_load_by_processing_job_name>
    <Function test_create>
    <Function test_log_parameter>
    <Function test_enter>
    <Function test_cm>
    <Function test_cm_fail>
    <Function test_enter_sagemaker_job>
    <Function test_log_parameters>
    <Function test_log_input>
    <Function test_log_output>
    <Function test_log_metric>
    <Function test_log_metric_attribute_error>
    <Function test_log_metric_attribute_error_warned>
    <Function test_log_output_artifact>
    <Function test_log_input_artifact>
    <Function test_log_inputs_error>
    <Function test_log_outputs>
    <Function test_log_multiple_input_artifact>
    <Function test_log_multiple_output_artifact>
    <Function test_log_pr_curve>
    <Function test_resolve_artifact_name>
    <Function test_artifact_uploader_init>
    <Function test_artifact_uploader_upload_artifact_file_not_exists>
    <Function test_artifact_uploader_s3>
    <Function test_guess_media_type>
    <Function test_lineage_artifact_tracker>
    <Function test_convert_dict_to_fields>
    <Function test_convert_data_frame_to_values>
    <Function test_convert_data_frame_to_fields>
    <Function test_convert_df_type_to_simple_type>
    <Function test_log_table_both_specified>
    <Function test_log_table_neither_specified>
    <Function test_log_table_invalid_values>
    <Function test_log_table>
    <Function test_log_roc_curve>
  <Module test_training_job.py>
    <Function test_search>
  <Module test_trial.py>
    <Function test_load>
    <Function test_create>
    <Function test_create_with_tags>
    <Function test_create_no_name>
    <Function test_create_with_trial_components>
    <Function test_add_trial_component>
    <Function test_add_trial_component_from_trial_component_summary>
    <Function test_remove_trial_component>
    <Function test_remove_trial_component_from_trial_component_summary>
    <Function test_remove_trial_component_from_tracker>
    <Function test_list_trials_without_experiment_name>
    <Function test_list_trials_with_experiment_name>
    <Function test_list_trials_with_trial_component_name>
    <Function test_search>
    <Function test_boto_ignore>
    <Function test_delete>
    <Function test_delete_all_with_incorrect_action_name>
    <Function test_delete_all>
    <Function test_delete_all_fail>
  <Module test_trial_component.py>
    <Function test_create>
    <Function test_create_with_tags>
    <Function test_load>
    <Function test_list>
    <Function test_list_empty>
    <Function test_list_trial_components_call_args>
    <Function test_search>
    <Function test_save>
    <Function test_delete>
    <Function test_delete_with_force_disassociate>
    <Function test_list_trials>
    <Function test_boto_ignore>
  <Module test_utils.py>
    <Function test_sagemaker_client_endpoint_env_set>
    <Function test_get_or_create_default_bucket_bucket_already_owned>
    <Function test_get_or_create_default_bucket_operation_aborted>
    <Function test_get_or_create_default_bucket_too_many_buckets>
    <Function test_get_or_create_default_other_error>

====================================================================================== no tests ran in 2.79 seconds ======================================================================================
________________________________________________________________________________________________ summary _________________________________________________________________________________________________
  collect-tests: commands succeeded
  congratulations :)