ng.pipe_proc.zf() inverting alternate column values

[andrealorenzon]

zf() output matrix appears to have all even columns' values inverted, but only on the second dimension processing. The first dimension works perfectly.

Example:

Input matrix part:

code:

dict, data = ng.pipe_proc.zf(dict, data, zf=1, auto=True)

output:

Matrix is otherwise correctly zero-filled, and shape is updated accordingly.

[kaustubhmote]

That's strange. AFAIK, the only thing that can do this is the alt=True flag in the ft function, but this is unlikely to be the cause here. I did a few quick tests on a 2d dataset, both in the direct and the indirect dimensions, but was not able to reproduce this. Can you post the full processing script to help debug this?

[andrealorenzon]

Here it is: every glue_data_amx[n] is a tuple of (dict,data). n=0 is raw fid file read.

# validated
glue_data_amx[1] = ng.pipe_proc.sp(glue_data_amx[0][0], 
                                   glue_data_amx[0][1], 
                                   off=0.5, end=0.95, pow=2, c=0.5)

# validated
glue_data_amx[2] = ng.pipe_proc.zf(glue_data_amx[1][0], 
                                   glue_data_amx[1][1], 
                                   zf=1, auto=True)

# validated
glue_data_amx[3] = ng.pipe_proc.ft(glue_data_amx[2][0], 
                                   glue_data_amx[2][1])

# digital filter fix, validated
N=np.fromfile("./filter/full_pipeline/amx/test0.fid", "float32", 512)[40]
glue_data_amx[3] = ng.pipe_proc.ps(glue_data_amx[3][0], 
                                   glue_data_amx[3][1], 
                                   p0=0, p1=-360*N,)
# validated
glue_data_amx[4] = ng.pipe_proc.ps(glue_data_amx[3][0], 
                                   glue_data_amx[3][1],
                                   p0=-41.5, p1=-8.0)
# validated
glue_data_amx[5] = ng.pipe_proc.tp(glue_data_amx[4][0], 
                                   glue_data_amx[4][1], hyper=True) 
# validated
glue_data_amx[6] = ng.pipe_proc.sp(glue_data_amx[5][0], 
                                   glue_data_amx[5][1], 
                                   off=0.5, end=0.95, pow=2, c=1)

# this is the failing step. [7] have inverted even columns (both in real and imaginary parts)
glue_data_amx[7] = ng.pipe_proc.zf(glue_data_amx[6][0],
                                   glue_data_amx[6][1],
                                   auto=True
                                  ) 
# this hardcode-fixes the issue:
glue_data_amx[7][1][...,1::2] *= -1        

# this is the rest of the pipeline i need to validate...        
glue_data_amx[8] = ng.pipe_proc.ft(glue_data_amx[7][0], 
                                   glue_data_amx[7][1],
                                  )

glue_data_amx[9] = ng.pipe_proc.ps(glue_data_amx[8][0], 
                                   glue_data_amx[8][1], 
                                  p0=-87.0, 
                                  p1=173.,)

glue_data_amx[10] = ng.pipe_proc.tp(glue_data_amx[9][0], 
                                    glue_data_amx[9][1])

I'm validating against a corresponding nmrpipe pipeline, that is

nmrPipe -in test.fid \
| nmrPipe -fn SP -off 0.5 -end 0.95 -pow 2 -elb 0.0 -glb 0.0 -c 0.5 \
| nmrPipe -fn ZF -zf 1 -auto \
| nmrPipe -fn FT -verb \
| nmrPipe -fn PS -p0 -41.5 -p1 -8.0 -di \
| nmrPipe -fn EXT -x1 0% -xn 100% -sw \
| nmrPipe -fn TP \
| nmrPipe -fn SP -off 0.5 -end 0.95 -pow 2 -elb 0.0 -glb 0.0 -c 1.0 \
| nmrPipe -fn ZF -zf 1 -auto \
| nmrPipe -fn FT -alt -verb \
| nmrPipe -fn PS -p0 -87.0 -p1 173.0 -di \
| nmrPipe -fn TP \
  -out test.ft2 -ov

Plots: orange is the computed data, blue is the data I'm validating against. Output without correction:

Output with correction:

[andrealorenzon]

For unknown reasons, here is a fully validated pipeline:

glue_data_dmx[1] = ng.pipe_proc.sp(glue_data_dmx[0][0], 
                                   glue_data_dmx[0][1], 
                                   off=0.5, end=0.95, pow=2, c=0.5)

glue_data_dmx[2] = ng.pipe_proc.zf(glue_data_dmx[1][0], 
                                   glue_data_dmx[1][1], 
                                   zf=1, auto=True)

glue_data_dmx[3] = ng.pipe_proc.ft(glue_data_dmx[2][0], 
                                   glue_data_dmx[2][1])

glue_data_dmx[4] = ng.pipe_proc.ps(glue_data_dmx[3][0], 
                                   glue_data_dmx[3][1],
                                   p0=-41.5, p1=-8.0)

glue_data_dmx[4] = ng.pipe_proc.di(glue_data_dmx[4][0], 
                                   glue_data_dmx[4][1])

glue_data_dmx[5] = ng.pipe_proc.tp(glue_data_dmx[4][0], 
                                   glue_data_dmx[4][1]) 

glue_data_dmx[6] = ng.pipe_proc.sp(glue_data_dmx[5][0], 
                                   glue_data_dmx[5][1], 
                                   off=0.5, end=0.95, pow=2, c=1)

glue_data_dmx[7] = ng.pipe_proc.zf(glue_data_dmx[6][0],
                                   glue_data_dmx[6][1],
                                   auto=True
                                  )

glue_data_dmx[7][1][...,1::2] *= -1

glue_data_dmx[8] = ng.pipe_proc.ft(glue_data_dmx[7][0], 
                                   glue_data_dmx[7][1],
                                  )

glue_data_dmx[9] = ng.pipe_proc.ps(glue_data_dmx[8][0], 
                                   glue_data_dmx[8][1], 
                                   p0=-87.0, 
                                   p1=173.,)

glue_data_dmx[9] = ng.pipe_proc.di(glue_data_dmx[9][0], 
                                   glue_data_dmx[9][1])

glue_data_dmx[10] = ng.pipe_proc.tp(glue_data_dmx[9][0], 
                                    glue_data_dmx[9][1])

[kaustubhmote]

I see that you have a -alt flag for the second ft in your nmrpipe script. This essentially does what glue_data_dmx[7][1][...,1::2] *= -1 does. So you can skip the manual setting and add alt=True keyword to your second ft function, the datasets should match. Based on your plots, you seem to also be comparing the 8th entry, which is after ft. So the above changes should fix this. In any case, I am unable to reproduce the behaviour where applying only the zf function give sign alternation.

[andrealorenzon]

Ok, thank you. That solved the issue.