search

aws-samples / sagemaker-run-notebook

Tools to run Jupyter notebooks as jobs in Amazon SageMaker - ad hoc, on a schedule, or in response to events
Apache License 2.0
142 stars 70 forks source link

Are Glue Jobs/DevEndpoints Supported with Sparkmagic kernel? #26

Open OElesin opened 3 years ago

OElesin commented 3 years ago

Hello team,

Do you currently support AWS Glue Jobs/DevEndpoints? This will surely bring this product close to Databricks?

tomfaulhaber commented 3 years ago

@OElesin I don't understand exactly what you're asking here.

Are you asking if you can use this tool to schedule a notebook that will access a Glue dev endpoint? I think the answer is yes, but I haven't tried it.

Or are you asking whether you can use this tool to run a notebook using a Glue dev endpoint rather than a SageMaker Processing Job. The answer to that is "no" but it might be possible to modify this tool to do that. I haven't looked at whether that would work, but it's an interesting idea.

Or did you mean something else?

OElesin commented 3 years ago

Are you asking if you can use this tool to schedule a notebook that will access a Glue dev endpoint? I think the answer is yes, but I haven't tried it. This is exactly my question rather than using EMR.