ledatelescope / bifrost

A stream processing framework for high-throughput applications.
BSD 3-Clause "New" or "Revised" License
66 stars 29 forks source link

How to output the data that input type int as the type of the np.float32 in a file in the TransformBlock subclass and save it to the file. #122

Closed zyfcharm closed 5 years ago

zyfcharm commented 6 years ago

Dear Scholar, I want to convert the data type in the file from unit16 to np.float32 in the TransformBlock subclass and run it in the file in the TransformBlock subclass, but I found the data type of ospan In the on_data method of TransformBlock,must be the same as ispan, leading to failure all the time, the following is my code, Sincerely request your guidance.Thank you。

class Data_DecodeBlock(bfp.TransformBlock): def init(self, iring, *args, *kwargs): super(Data_DecodeBlock, self).init(iring, args, **kwargs)

def on_sequence(self, iseq):
    ihdr = iseq.header
    ohdr = deepcopy(ihdr)
   return ihdr

def on_data(self, ispan,ospan):
    in_nframe = ispan.nframe
    out_nframe = in_nframe
   ospan.data[...] = ispan.data.astype(np.float32)

if name == "main": filenames1 = (glob.glob('generate_2_64_8.bin.out')) b_read1 = BinaryFileReadBlock(filenames1, 64,64 , 'u16', core=0) b_decode = Data_DecodeBlock(b_read1,core = 2) b_write = BinaryFileWriteBlock(b_decode, core=4) pipeline = bfp.get_default_pipeline() print pipeline.dot_graph() pipeline.run()

thanks, yanfei

telegraphic commented 6 years ago

Hi @zyfcharm ,

in bifrost, the dtype and shape of the buffers that connect blocks together are setup using the header. Specifically, if you change the shape / dtype / rearrange axes etc, you need to update the _tensor inside your on_sequence method.

There's not enough documentation on this yet. But, here's a quick overview where we touch on it: http://ledatelescope.github.io/bifrost/your-first-blocks.html#the-tensor-dict

Basically, your code needs an extra line:

def on_sequence(self, iseq):
    ihdr = iseq.header
    ohdr = deepcopy(ihdr)
    ohdr['_tensor']['dtype'] = 'f32'
   return ihdr

def on_data(self, ispan,ospan):
    in_nframe = ispan.nframe
    out_nframe = in_nframe
   ospan.data[...] = ispan.data.astype(np.float32)

However, you can probably just use the in-built bifrost.blocks.quantize to convert the data! Here's the docstring:

def quantize(iring, dtype, scale=1., *args, **kwargs):
    """Apply a requantization of bit depth for the data.
    Args:
        iring (Ring or Block): Input data source.
        dtype: Output data type or number of bits.
        scale (float): Scale factor to apply before quantizing.
        *args: Arguments to ``bifrost.pipeline.TransformBlock``.
        **kwargs: Keyword Arguments to ``bifrost.pipeline.TransformBlock``.
    **Tensor semantics**::
        Input:  [...], dtype = [c]f32, space = SYSTEM
        Output: [...], dtype = any (complex) integer type, space = SYSTEM
    Returns:
        QuantizeBlock: A new block instance.
    """
return QuantizeBlock(iring, dtype, *args, **kwargs)
zyfcharm commented 5 years ago

@telegraphic : I am very glad to receive your reply.I sincerely appreciate your guidance and let me find a solution to this problem. Thank you.