search

coiled / benchmarks

BSD 3-Clause "New" or "Revised" License
28 stars 17 forks source link

Use `get_spark_cluster` to start spark cluster #1499

Closed fjetter closed 5 months ago

fjetter commented 6 months ago

There's a dedicated coiled API for launching a spark cluster on Coiled. This is mostly just a wrapper around the normal cluster but it is setting some networking config that allows us to access the dashboard.

There is also no reason to have the spark specific stuff in the general top level conftest

ntabris commented 6 months ago

Unless you're using _open_debug_ports (which I'd recommend against), there's no difference from using a regular cluster (which I'd recommend).

fjetter commented 6 months ago

from using a regular cluster

I was having difficulties with connecting to the spark dashboard. I thought this was opening additional ports

ntabris commented 6 months ago

I was having difficulties with connecting to the spark dashboard. I thought this was opening additional ports

Not unless you pass _open_debug_ports=True.

If you're having trouble connecting to spark dashboard, it's probably some other issue... happy to look at this if you have more info or want to look together.