search

3dgeo-heidelberg / py4dgeo

py4dgeo - A Python library for change analysis in 4D point clouds
https://py4dgeo.readthedocs.io
MIT License
66 stars 10 forks source link

add_epochs for large number of initial epochs #159

Closed kathapand closed 2 years ago

kathapand commented 2 years ago

I need to add a large number of epochs (> 2,500) to a SpatiotemporalAnalysis object for the extraction of 4D objects-by-change. Adding them one by one takes too much time, due to rearranging of the array (known behavior): at the 500th epoch rearranging takes already 180 s.

As all epochs are already available (no online acquisition and analysis), I can add all epochs, by compiling a list of all generated epochs: 

# loop over epochs and add (except the reference epoch)
epoch_batch = []
for idx, row in df_timeseries.iterrows():
    fname_cloud = row['fname_timestring']
    if fname_cloud == fname_reference_cloud:
        continue
    subdir_daystring = fname_reference_cloud.split('_')[0]
    f_cloud = f'{data_dir}/{subdir_daystring}/{fname_reference_cloud}.laz'

    trafomat = read_corresponding_trafofile(f_cloud)
    cloud = read_las(f_cloud, trafomat = trafomat)

    epoch = py4dgeo.epoch.as_epoch(cloud)
    epoch.timestamp = row['timestamp']

    epoch_batch.append(epoch)

    #if idx > 5:
    #    break

analysis.add_epochs(*epoch_batch)
kathapand commented 2 years ago

The question/issue has become obsolete. I am here posting the result for documentation and potentially future considerations:

APPROACH:

THE CODE:

# loop over epochs and add (except the reference epoch) [num_epochs = 2967]
batchsize = 500
b = 0
epoch_batch = []
num_epochs = len(df_timeseries)
for idx, row in df_timeseries.iterrows():
    print(f'Compiling epoch [{idx}/{num_epochs}]')       
    fname_cloud = row['fname_timestring']
    if fname_cloud == fname_reference_cloud:
        continue
    subdir_daystring = fname_reference_cloud.split('_')[0]
    f_cloud = f'{data_dir}/{subdir_daystring}/{fname_reference_cloud}.laz'

    trafomat = read_corresponding_trafofile(f_cloud)
    cloud = read_las(f_cloud, trafomat = trafomat)

    epoch = py4dgeo.epoch.as_epoch(cloud)
    epoch.timestamp = row['timestamp']

    epoch_batch.append(epoch)

    b+=1

    if b%batchsize == 0:
        print('Adding batch of epochs...')
        analysis.add_epochs(*epoch_batch)
        epoch_batch = []

THE OUTPUT:

[2022-06-18 10:39:57][INFO] Finished in 0.4481s: Adding epoch 500/500 to analysis object
[2022-06-18 10:39:57][INFO] Starting: Rearranging space-time array in memory
[2022-06-18 10:42:23][INFO] Finished in 146.0912s: Rearranging space-time array in memory
[2022-06-18 10:42:23][INFO] Starting: Updating disk-based analysis archive with new epochs
kathapand commented 2 years ago

Fifth batch (>2000) epochs cannot be added:

File H:\conda_envs\py4dgeo_dev\lib\site-packages\py4dgeo\segmentation.py:384, in SpatiotemporalAnalysis.add_epochs(self, *epochs)
    380     # Load the uncertainty array and append new data
    381     uncertainty_file = os.path.join(tmp_dir, uncertainty_filename)
    382     write_func(
    383         uncertainty_file,
--> 384         np.concatenate(
    385             (self.uncertainties, np.column_stack(tuple(new_uncertainties))),
    386             axis=1,
    387         ),
    388     )
    390 # Dump the updated files into the archive
    391 with logger_context("Updating disk-based analysis archive with new epochs"):

File <__array_function__ internals>:180, in concatenate(*args, **kwargs)

MemoryError: Unable to allocate 66.4 GiB for an array with shape (709791, 2512) and data type [('lodetection', '<f8'), ('spread1', '<f8'), ('num_samples1', '<i8'), ('spread2', '<f8'), ('num_samples2', '<i8')]
dokempf commented 2 years ago

I have done similar studies where I added constant size batches of epochs and observed dramatically increasing processing times due to the rearranging in memory. I will try to check whether np.concatenate/np.column_stack does something inefficient here by trying a C++ implementation of the rearranging step.

dokempf commented 2 years ago

As far as the out-of-memory issue is concerned: Uncertainties take 5 times more memory than distances. As you did not store them in your previous code base, you might not be able to process the same datasets with py4dgeo (on the same machine). Not sure what can be done about that - unless storing the uncertainties is not a requirement.

kathapand commented 2 years ago

That makes sense... but it is right to store the uncertainties, even if they are not always required for an analysis step.

(A) Maybe when the memory policy is set to "MINIMAL" the arrays can be loaded and extended separately? This increases the capacity - because for 4D-OBC extraction at the moment we only use the distances (in the future will be an option to include uncertainties).

(B) As a general issue of handling the huge 3D time series: I guess we need an option to perform the analysis for a part of the data, e.g. by defining a timespan. Thinking on long-term monitoring, we will add evermore epochs but analyze looking back only a certain timespan (not always the full period since start).

dokempf commented 2 years ago

I tried a C++ implementation of the rearranging step and it turns out to not be a significant improvement over the status quo. Closing this now.