Create database on the fly for inference

[jonasteuwen]

Is your feature request related to a problem? Please describe. Right now you need to create a database to do inference

Describe the solution you'd like On the fly creation (e.g. in memory) for inference.

[YoniSchirris]

My plan is:

• create a new DataDescription class, namely OnTheFlyDataDescription (or something that sounds nicer), that includes only the parameters required for an in-memory database

class DataDescription(BaseModel):
    !! REMOVE -- here to generate a mask !! mask_label: Optional[str] = None
    !! REMOVE -- not training, just inferring!! mask_threshold: Optional[float] = None 
    !! REMOVE -- only input is WSI !! roi_name: Optional[str] = None
    !! REMOVE -- not sure what this is? !! num_classes: NonNegativeInt
    == KEEP  -- need a root dir for WSIs == data_dir: Path 
    !! REMOVE -- we'll generate this on the fly !!manifest_database_uri: str
    ?? NOT SURE -- i dont know what a manifest is ?? manifest_name: str
    !! REMOVE -- not using splits, just do everything !! split_version: str
    !! REMOVE -- notu sing any, just generating !! annotations_dir: Path
    ?? NOT SURE? Remove? not training? ?? training_grid: GridDescription
    == KEEP -- we need to know how to infer == inference_grid: GridDescription
    !! REMOVE !! index_map: Optional[Dict[str, int]]
    !! REMOVE !!remap_labels: Optional[Dict[str, str]] = None
    !! REMOVE !! use_class_weights: Optional[bool] = False
    !! REMOVE !! convert_mask_to_rois: bool = True
    !! SET TO FALSE !! use_roi: bool = True
    ?? NOT SURE? ??apply_color_profile: bool = False

So i guess we get something like

class OnTheFlyDataDescription(BaseModel):
    data_dir: Path # root dir for WSIs
    wsi_glob_pattern: str # add a glob pattern to find all WSIs
    inference_grid: GridDescription # Inference details

Then, in inference.py, before running inference, there'll be a check for the __target__ of the data_description config. If it's something on-the-fly, we'll generate a minimal database in memory, set the config properly, and do inference.

I need to check how to do the in-memory db, but if you agree with this i can start this tomorrow.

[YoniSchirris]

See 4a524bcbc18d26cbad3bab8a98abd8149ff9793b