search

pycroscopy / SciFiReaders

Tools for extracting data and metadata from scientific data files
https://pycroscopy.github.io/SciFiReaders/about.html
MIT License
13 stars 13 forks source link

Problem reading multiple files using NSIDReader #69

Open badarirao opened 2 years ago

badarirao commented 2 years ago

Hello, I am able to successfully read one hdf5 file using NSIDReader and store it in a variable. However, when I read another file and store it in another variable, somehow the 1st variable data is also replaced by the new file. Could you please help me with this?

example: dset1 = NSIDReader("data1.h5").read() --> dset1 consists of data from data1.h5 dset2 = NSIDReader("data2.h5").read() --> now both dset1 and dset consists of data from data2.h5.

How do I prevent dset1 from being overwritten?

I have tried the NSIDReader from both SciFiReaders and pyNSID modules, with the same result.

badarirao commented 2 years ago

Found a temparory solution for this: Make copy of each element in the list. dset1 = NSIDReader("data1.h5").read() dset1copy = [] for data in dset1: dset1copy.append(data.copy()) dset2 = NSIDReader("data2.h5").read() dset2copy = [] for data in dset2: dset2copy.append(data.copy())

This seems to ensure the two data remain different.

ramav87 commented 2 years ago

Thanks for letting us know. The expected usage of readers is one reader per file. In your case you've instantiated a single object and (I believe) the same object is being used to read a new file resulting in it being overwritten. It would be better to write it as follows:

reader1 = NSIDReader("data1.h5") dset1 = reader1.read()

reader2 = NSIDReader("data2.h5") dset2 = reader2.read()

This also is the better way because it allows you to close files.

badarirao commented 2 years ago

@ramav87 I just checked your solution, but the situation does not change. Strangely, the memory address of the two objects seems to be different, but the data is overwritten. Only copy() seems to work.

ramav87 commented 2 years ago

I think it might be an issue with your specific interptretor. I tested this on Google Colab and it is functional:

https://colab.research.google.com/drive/1n3iOGdUOff1t1Jbmq84GaIV-JK03lweP?usp=sharing

Please take a look and let me know.

badarirao commented 2 years ago

@ramav87 Okay, I checked in Google Colab, and your code seems to be working. Upon deeper inspection, I found that this is the case even in my local machine. The problem is with its metadata, which is being overwritten. So, the metadata of dset1 is overwritten by that of dset2, whereas the actual data is not changed.