search

mlexchange / mlex_dlsia_segmentation_prototype

Other
3 stars 3 forks source link

Added Tiled saving frame on the fly. #12

Closed TibbersHao closed 6 months ago

TibbersHao commented 6 months ago

This PR covers 1 major feature upgrade and several minor changes:

Feature Upgrade:

Added functions to pre-allocate array space on Tiled Client, and saving single frame on the fly during inference.

Minor Changes:

  1. Now 2 uids will be taken from yaml file: uid_save for model and metric saving, and uid_retrieve for model retrieval.
  2. Added conversion codes to convert str of lists to np array, list, or float tensor when needed.

To Test:

  1. Modify uri and api_key in example yaml files
  2. run make file commands
  3. In order to test full inference saving, select correct uid_retrieve for previous trained models, and comment out mask_tiled_uri, mask_tiled_api_key (or set them to null in yaml).

Note:

  1. It would be worthwhile to try at least 1 full inference to see if all slices have been saved properly, eg. a very small MSDNet. In my test case, I encountered unstable connection to public tiled server due to spiky internet, and apparently there is no mechanism in code for recovery in such situation. Don't know if this would be a concern for the beam time. As a reference, the error code I got was: httpx.ReadError: [Errno 54] Connection reset by peer
  2. The SMSNet Ensemble methods should still be tested out as this hasn't been well addressed in my last PR due to time concern.
TibbersHao commented 6 months ago

@taxe10 @Wiebke most of changes have been addressed with the recent push, please take a look.

I will be waiting for the uid discussion to see where we want to go, and we are close to wrap up this PR.