kevinjqliu commented 1 month ago

Description

This PR moves the docker-compose-jupyter.yml file (and the notebooks/ directory), formerly in the top-level directory, into the getting-started/spark/ folder.

The purpose is to unify the "getting started" guides into the same directory.

Fixes #110

Type of change

Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] Documentation update
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] This change requires a documentation update

How Has This Been Tested?

docker-compose -f getting-started/spark/docker-compose.yml up

Open the SparkPolaris.ipynb Jupyter notebook Grab the root principal credentials from the Polaris service and replace in the notebook cell. Run all cells in notebook

Checklist:

Please delete options that are not relevant.

[ ] I have performed a self-review of my code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] If adding new functionality, I have discussed my implementation with the community using the linked GitHub issue

kevinjqliu commented 1 month ago

I want to make sure this is something we want to do before proceeding to add more to the PR

cc @collado-mike / @flyrain

flyrain commented 1 month ago

Make sense to me. Thanks @kevinjqliu! Do we have any doc for its usage? We may add doc if not.

kevinjqliu commented 1 month ago

@flyrain yep i'll have a README in here, similar to the trino one

flyrain commented 1 month ago

Sounds good. We will need these doc to be in the Polaris doc site, like this https://polaris.apache.org/docs/overview/. I couldn't find Trino's doc there, this may involve doc publish and link. cc @jbonofre

kevinjqliu commented 1 month ago

I see, this is the README for trino. I'll add a similar README for spark.

As a follow-up, we can change the Polaris doc to refer to these guides https://polaris.apache.org/docs/quickstart

collado-mike commented 1 month ago

This looks good to me. We should change the name of the compose file to just docker-compose.yml so we don't have to specify the filename in the command line :)

kevinjqliu commented 1 month ago

@collado-mike makes sense, will do.

I have a question on slack about unable to assume the role arn:aws:iam::631484165566:role/datalake-storage-integration-role in the notebook, do you mind taking a look?

kevinjqliu commented 1 month ago

r? @flyrain @RussellSpitzer @collado-mike

Also opened #319 to update the Polaris doc site once this is merged.

kevinjqliu commented 1 month ago

md check intermittently shows https://redocly.com/docs/cli/installation as 400 error, weird

flyrain commented 1 month ago

md check intermittently shows https://redocly.com/docs/cli/installation as 400 error, weird

It's OK to remove the link for now since we’re transitioning to Hugo.

kevinjqliu commented 1 month ago

@flyrain just had to run the CI a few times, it's unrelated to this change

kevinjqliu commented 3 weeks ago

Thanks for the review @flyrain, addressed your comments

flyrain commented 3 weeks ago

We cannot merge any PR until #374 is merged.

kevinjqliu commented 3 weeks ago

Thanks for the heads up, I'll rebase once that PR's merged

kevinjqliu commented 3 weeks ago

@flyrain took your advice, moved getting-started/spark/create-polaris-catalog.sh logic into the jupyter notebook. Also rebased off latest main. I think this PR is good to go. Please take a look!

flyrain commented 3 weeks ago

Thanks a lot for working on it, @kevinjqliu! Thanks all for the review.

apache / polaris

Spark Jupyter getting started docker compose #295

Description

Type of change

How Has This Been Tested?

Checklist: