search

quinngroup / dr1dl-Dask

Dictionary Learning in Dask
0 stars 0 forks source link

Command Line arguments #4

Closed crumleyc closed 4 years ago

crumleyc commented 4 years ago

In the Pyspark implementation, there were two arguments that I really had to do anything with.

  1. Partitions --> Chunks
    • Dask Chunks
    • If an int x is supplied then each chunk will be (x,x)
    • A tuple of things can be applied as well so give Dask a more specific structure of all the chunks
    • I changed the default value to 'auto' because that is the default for Dask.
  2. execmem
    • I can't really decide if this is needed. I have checked the parameters of the client and there isnt a way to establish how much memory a worker should have. Plus when run locally, Dask sets its regardless in the LocalCluster class.