search

equinor / segyio

Fast Python library for SEGY files.
Other
469 stars 213 forks source link

How to load a SEGY file that does not fit into memory? #529

Closed jcfaracco closed 1 year ago

jcfaracco commented 2 years ago

Hello guys,

I'm trying to load a SEG-Y file using segyio, but the file is larger than the memory I have. So, I'm using the following code:

import segyio
import dask.array as da

filename = "/foo/bar.sgy"

with segyio.open(filename) as segyf:
    segyf.mmap()
    cube = da.from_array(segyf.trace.raw)

It returns an error saying that the attribute shape is an integer instead of a tuple.

TypeError: object of type 'int' has no len()

Maybe, it should be nice to extend the property shape to the Trace class to support dask array also.

Any other thoughts? Using SEGYSAK is not an option here. I need to use Dask.

ErlendHaa commented 2 years ago

Hi,

The reason for your error is that da.from_array expects a numpy ndarray, while you supply it a segyio.RawTrace. Trace and RawTrace doesn't read any traces from disk until you explicitly ask for them: f.trace[1:2]. Once you've read a trace, or a set of them, you can pass it directly to dask.

If you don't want or need the full cube in memory, read individual traces (f.trace[n]) or full lines (e.g.f.iline[no]) on demand.

ErlendHaa commented 1 year ago

I'm gonna close this one now. Feel free to reopen if you still have troubles