search

googledatalab / datalab

Interactive tools and developer experiences for Big Data on Google Cloud Platform.
Apache License 2.0
975 stars 249 forks source link

Executing datalab notebook on composer airflow #2107

Open OrielResearchCure opened 5 years ago

OrielResearchCure commented 5 years ago

Hello,

I am using datalab for my my product development and access all Google Cloud Platform resources that I need. I would like to take it into composer to allow a scheduled execution of the datalab notebook code. Could you please let me know if anyone has experience with that. I have see that Composer has a built in support for python scripts (https://cloud.google.com/composer/docs/how-to/using/writing-dags) but not for datalab notebooks. Please let me know if there a simple example out there that I can use as a reference.

I can convert the datalab notebook to python scripts, but I am not sure that it will support the datalab API that I occasionally use, like storage or the magic symbol, like ! for bash execution.

Many thanks, eilalan

yebrahim commented 5 years ago

@rajivpb did some work around Composer, perhaps he can provide some guidance here.