Batch size argument for inference does not work as expected

[delhomer]

The batch size parameter does not seem to have any impact on the system resource usage. One could expect that providing a smaller batch size would kind of "save" memory at the price of longer processing times.

However it does not fit this goal:

• batch_size = 1

  
  11:12 $ time deepo postprocess -D tanzania -b 1 -i grid_034 -s 384
  Using TensorFlow backend.
  2020-05-06 11:13:04,709 :: INFO :: postprocess :: main : Raw image size: 3850, 3860
  2020-05-06 11:13:04,711 :: INFO :: postprocess :: main : The image will be splitted into 121 tiles
  2020-05-06 11:13:07,004 :: INFO :: postprocess :: get_trained_model : Model weights have been recovered from ./data/tanzania/output/semseg/checkpoints/best-model-384.h5
  2020-05-06 11:13:07,004 :: INFO :: postprocess :: main : Predict labels for input images...
  121/121 [==============================] - 134s 1s/step
  2020-05-06 11:15:21,512 :: INFO :: postprocess :: main : Labelled image dimension: 3860, 3850

real 2m24,930s user 6m45,306s sys 0m12,090s

- `batch_size = 20`

11:20 $ time deepo postprocess -D tanzania -b 20 -i grid_034 -s 384 Using TensorFlow backend. 2020-05-06 11:23:46,304 :: INFO :: postprocess :: main : Raw image size: 3850, 3860 2020-05-06 11:23:46,306 :: INFO :: postprocess :: main : The image will be splitted into 121 tiles 2020-05-06 11:23:50,833 :: INFO :: postprocess :: get_trained_model : Model weights have been recovered from ./data/tanzania/output/semseg/checkpoints/best-model-384.h5 2020-05-06 11:23:50,833 :: INFO :: postprocess :: main : Predict labels for input images... 121/121 [==============================] - 158s 1s/step 2020-05-06 11:26:29,071 :: INFO :: postprocess :: main : Labelled image dimension: 3860, 3850

real 2m55,432s user 6m47,825s sys 0m47,111s

The running time is comparable, while the memory slightly increases with the batch size (I have some RAM troubles with `batch_size > 20`, the sys time is even larger due to memory swapping...).

As a side topic, whatever the batch size is, all my CPUs are full during the prediction. This is maybe a `keras` feature when feeding a model with a `np.array`. I will try to introduce generators to have an insight of an alternative way of doing.

[delhomer]

No real improvement with generator, all the CPUs are active as well, and the measured time is not really better than with np.array inputs.

There are two additional drawbacks regarding the generators (one uses the keras API, and flow_from_directory, see https://github.com/Oslandia/deeposlandia/blob/cd56791f4d38d0857313649082d176b979c27902/deeposlandia/generator.py#L144) :

• the images are shuffled, hence one has major troubles for rebuilding the original image ;
• flow_from_directory means that all the images within the targetted directory will be evaluated, this being incompatible with the current folder management in Deeposlandia.

I close this issue for now, we might open it again if the question comes back later.