Propose to update the defaults in the template configuration

[noklam]

Hi, thank you for creating this plugin. I have tried to play around with it and successfully run my pipeline on SageMaker.

Context

This is mainly from the perspective of someone who just wants to try out the plugin and see how it works. I try to document the challenges that I encountered and I hope this will smooth out the onboarding process a bit.

Proposed Changes

• Update the default so it will work without any changes
  • Change machine type to ml.t3.medium, this is supported by the free-tier while the ml.t5.large.
  • __default__ should be made the default since the current template wouldn't work. I know it's mentioned in the comment but it is nicer to have a ready-to-run default.
  • There are not many docs about the setup for AWS, I know it's not exactly the responsibility of this plugin, but I think the user will appreciate some guide on setting up S3 + IAM role + ECR repository + SageMaker + SageMaker Studio profile etc.
  • Populate a link to the corresponding experiment automatically - I am not sure if is this possible, but when I first run the pipeline I struggle to see my pipeline until later I find out I need to create a profile and check my pipeline in Studio. It would be nice if there is a clickable link in the terminal.

Thanks for creating the plugin, the video is also quite useful as it provides extra instructions that aren't covered in the documentation.

[marrrcin]

Thanks @noklam for your suggestions! I'm really happy that the plugin worked for you 🙂

The changes you've suggested will definitely improve the end user experience.

Populate a link to the corresponding experiment automatically - I am not sure if is this possible, but when I first run the pipeline I struggle to see my pipeline until later I find out I need to create a profile and check my pipeline in Studio. It would be nice if there is a clickable link in the terminal.

Do you mean something like displaying the URL for the started pipeline in the terminal (like we do for Azure ML for example)? I was looking for a way to show it somehow, but I was not able to find an easy way to do this yet - it might required some more research + I'm not sure whether SageMaker supports deep linking at all 🤔

[noklam]

@marrrcin Yeah, I am thinking of a link that will lead users to the experiment page automatically. It's great that you have already considered that, I understand it could be tricky to figure out how to implement it.

[marrrcin]

@noklam , please also modify the unit tests:

  _____________ test_should_generate_pipeline_with_processing_steps ______________

  context_mock = <MagicMock name='KedroContext' id='140018508686096'>
  no_mlflow = None

      @patch("kedro.framework.project.pipelines", {"__default__": sample_pipeline})
      @patch("kedro.framework.context.KedroContext")
      def test_should_generate_pipeline_with_processing_steps(context_mock, no_mlflow):
          # given
          config = _CONFIG_TEMPLATE.copy(deep=True)
          generator = KedroSageMakerGenerator(
              "__default__", context_mock, config, is_local=False
          )

          # when
          pipeline = generator.generate()

          # then
          assert isinstance(pipeline.sagemaker_session, pipeline_context.PipelineSession)
  >       assert pipeline.name == "kedro-sagemaker-pipeline"  # a default one
  E       AssertionError: assert 'my-pipeline' == 'kedro-sagemaker-pipeline'
  E         - kedro-sagemaker-pipeline
  E         + my-pipeline

[noklam]

@marrrcin How do I run the test here? The CI doesn't seem to run correctly. I have some trouble setting the environment up when I do pip install . It has trouble finding the correct version of boto3

[marrrcin]

We're using Poetry in this repo, the setup is described here: https://kedro-sagemaker.readthedocs.io/en/latest/source/04_development.html

[noklam]

the test looks fine now, sonar is failing with missing token which I can't fix. @marrrcin