Closed avriiil closed 2 years ago
how large is the compute warehouse? this sounds like you might've used a smaller warehouse than I did. I'm not sure if the video shows what I used back then but I would simply suggest to bump that up.
Either way, it would be helpful if you can find out what takes that much time (e.g. a performance report). The dask snowflake connector doesn't perform any computations itself so I strongly suspect this is on snowflakes side and the most likely source is the warehous size
I've asked @pavithraes to run the code above on her local non-M1 machine to see if she can reproduce.
Here are my timings:
Confirming that this was an issue due to warehouse size and closing this issue.
I'm seeing some performance issues while reading / writing data using the
dask-snowflake
connector.Specifically, the
to_snowflake
call in the code below (taken from the coiled docs) sometimes runs in ~25 seconds and other times takes >1min.And the
read_snowflake
call in the code below ran in ~40s in @fjetter's original demo video but takes >4min. in my case.I'm running this from a notebook on an EC2 instance because the
snowflake.connector
still doesn't run on M1. I've asked @pavithraes to run the code above on her local non-M1 machine to see if she can reproduce.