set chunksize to some value greater than 1 in geojson conversion

[kaczmarj]

/[redacted]/envs/wsinfer/lib/python3.10/site-packages/wsinfer/write_geojson.py:112: TqdmWarning: Iterable length 4100 > 1000 but `chunksize` is not set. This may seriously degrade multiprocess performance. Set `chunksize=1` or more.
  process_map(func, csvs, max_workers=num_workers)

[swaradgat19]

@kaczmarj I could take this issue up! What would be the best chunk size for our case? I was looking at this StackOverflow answer regarding chunksizes. Going through some more documentation to see what will be best for our case

[kaczmarj]

great! thanks for the link to the stack overflow answer. according to that, inter-process communication time is an important consideration when choosing the chunk size. the thing is, the duration of the conversion greatly outweighs the duration of process switching.

i wonder if chunksize=1 would be best actually...

could you test a few and see which gives the best timing results? perhaps test 1, 4, 10, and 20.

[swaradgat19]

Sure. I will run some tests and will see what gives the best results.

[swaradgat19]

I tried out different chunksizes. Here is what I observed:

Exp 1:

• chunksize = 1
• Number of WSI = 12
• Time elapsed = 7.646886587142944

Exp 2:

• chunksize = 4
• Number of WSI = 12
• Time elapsed = 20.596678256988525

Exp 3:

• chunksize = 10
• Number of WSI = 12
• Time elapsed = 42.19792675971985

Exp 4:

• chunksize = 20
• Number of WSI = 12
• Time elapsed = 46.71947002410889

I believe using chunksize = 1 seems to be the best option.

[kaczmarj]

thanks @swaradgat19