robertu94 / libpressio

A library to abstract between different lossless and lossy compressors
Other
30 stars 11 forks source link

Installation failed following instructions #37

Open legendPerceptor opened 1 week ago

legendPerceptor commented 1 week ago

I want to test the following compressors with libpressio: TTHRESH, SZ3, SZ, ZFP, fpzip. I want to run the compression/decompression in both Python3 and C++.

I did the following commands to install them with spack. I just tried to install them on my personal Desktop computer (X86_64 architecture, Intel CPU with a Ubuntu 24.04 system).

spack env create icomp
spack env activate icomp
spack add libpressio-tools+tthresh ^libpressio+python+sz3+sz+zfp+fpzip
spack install

I got a bunch of errors as shown below. image

The whole build log can be found here.

I am not sure how spack is installing things, but I hope to change certain parts of pressio library and recompile it. I kind of need a development mode. It looks like the spack is using some built-in libpressio package to install things in the robertu94_packages.

Conceptually, I want to achieve the following idea.

import sz3
import tthresh
import zfp
import sz

def custom_load_data_function(data_path):
    """It can load only part of the data or a certain subset of the whole data file into memory"""
    pass

def save_data_to_file(data, path):
    pass

data = custom_load_data_function("./data.dat")

compressed_data = sz3.compress(data, config={...})
save_data_to_file(compressed_data, './compressed.sz3')
compressed_data = zfp.compress(data, config={...})
save_data_to_file(compressed_data, './compressed.zfp')
compressed_data = sz.compress(data, config={...})
save_data_to_file(compressed_data, './compressed.sz')
compressed_data = tthresh.compress(data, config={...})
save_data_to_file(compressed_data, './compressed.tthresh')

I wonder what is the best way for me to achieve this. Libpressio seems to be a good beginning step as it supports many compressors. It feels very hard to install and use though.

robertu94 commented 1 week ago

Thanks for the bug report @legendPerceptor . We have a fix for this issue, but we haven't released it while we complete some other required maintenance before SC24. Hopefully, I'll have an update for you soon.