search

pgrete / energy-transfer-analysis

BSD 3-Clause "New" or "Revised" License
6 stars 6 forks source link

Kernel Porting #13

Open aurora-cossairt opened 5 years ago

aurora-cossairt commented 5 years ago

7/2/2019

FYI copies of FlowAnalysis.py and run_analysis.py should (hopefully) be available here: https://github.com/pgrete/energy-transfer-analysis/tree/aurora

" Just wrote statistics for AbsRotU Traceback (most recent call last): File "/mnt/home/f0008575/src/energy-transfer-analysis/run_analysis.py", line 221, in FA.run_analysis() File "/mnt/home/f0008575/src/energy-transfer-analysis/FlowAnalysis.py", line 249, in run_analysis self.FT_momentum = self.FFT.forward(momentum, self.FT_momentum) File "/mnt/home/f0008575/yt-conda/lib/python3.7/site-packages/mpi4py_fft/mpifft.py", line 63, in call self.input_array[...] = input_array ValueError: could not broadcast input array from shape (3,32,256,256) into shape (32,256,256) "

aurora-cossairt commented 5 years ago

Additional errors found

Full output is saved in this repo as outfile.txt

When I run the programs, here are the print statements I ask for:

    delta_x = 1/self.res
    m = np.linspace(1, self.res/2, 1)
    k_lm = lm = np.zeros(int(self.res/2)-1)
    lm[:] = np.array(2*m[:]*delta_x)
    k_lm[:] = 1/lm[:]
    print("Before m shape")
    print("m shape: ", m.shape, "\nlm shape: ", lm.shape, "\nk_lm shape: ", k_lm.shape, file=sys.stderr)

    print("Between m shape and momentum")
    epsilon = all_filtered_momenta = all_filtered_rho = np.zeros(len(k_lm))
    momentum = self.rho * self.U
    print("Momentum is: ", momentum)
    print("Momentum shape is: ", momentum.shape)
    self.FT_momentum = self.FT_rho = newDistArray(self.FFT,rank=1)
    self.momentum_filtered = self.rho_filtered = newDistArray(self.FFT,rank=1)

So I should be seeing "Before m shape", "m shape", "Between m shape and momentum", and then "Momentum" and "Momentum shape" in that order.

Instead, I see this

In run_analysis
Finished KinEnSpecific
In Acc is not None
Just wrote statistics for AbsRotU
In Method B
Before m shape
Between m shape and momentum
Momentum is:  [[[[-2.57218443e-02 -1.05658602e-02  4.11360814e-03 ... -7.30447838e-02
    -5.70849826e-02 -4.12471236e-02]
   [-2.77948480e-02 -1.25384081e-02  2.21410829e-03 ... -7.54341405e-02
    -5.93471917e-02 -4.34150243e-02]
   [-2.96408314e-02 -1.43042372e-02  5.02051109e-04 ... -7.75418204e-02
    -6.13494920e-02 -4.53412177e-02]
   ...
   [-1.81538550e-02 -3.38080900e-03  1.09509011e-02 ... -6.41938143e-02
    -4.87268856e-02 -3.33130509e-02]
   [-2.09008492e-02 -5.98682529e-03  8.48027330e-03 ... -6.74221531e-02
    -5.17729695e-02 -3.61967936e-02]
   [-2.34236897e-02 -8.38236840e-03  6.20176291e-03 ... -7.03734106e-02
    -5.45598788e-02 -3.88415772e-02]]

  [[-3.22303794e-02 -1.71386351e-02 -2.48425943e-03 ... -7.92659632e-02
    -6.34602314e-02 -4.77104042e-02]
   [-3.46102854e-02 -1.94222252e-02 -4.67318036e-03 ... -8.19786649e-02
    -6.60403258e-02 -5.01780207e-02]
   [-3.67613281e-02 -2.14908542e-02 -6.66814686e-03 ... -8.44022737e-02
    -6.83513502e-02 -5.23996765e-02]
   ...
   [-2.37266175e-02 -9.00242398e-03  5.26630765e-03 ... -6.94178204e-02
    -5.41217900e-02 -3.88299611e-02]
   [-2.67854915e-02 -1.19280135e-02  2.48254501e-03 ... -7.29814546e-02
    -5.74983056e-02 -4.20364081e-02]
   [-2.96212750e-02 -1.46411510e-02 -9.96462510e-05 ... -7.62658852e-02
    -6.06123145e-02 -4.49963303e-02]]

  [[-3.87930516e-02 -2.37574276e-02 -9.16694597e-03 ... -8.54347349e-02
    -6.98229604e-02 -5.42142912e-02]
   [-4.14765947e-02 -2.63509730e-02 -1.16585059e-02 ... -8.84776038e-02
    -7.27270205e-02 -5.69944181e-02]
   [-4.39287941e-02 -2.87243978e-02 -1.39466288e-02 ... -9.12246542e-02
    -7.53548954e-02 -5.95188389e-02]
   ...
   [-2.93541865e-02 -1.46744992e-02 -4.66254810e-04 ... -7.45753775e-02
    -5.94853330e-02 -4.43517831e-02]
   [-3.27307702e-02 -1.79172793e-02 -3.56977910e-03 ... -7.84781234e-02
    -6.31981761e-02 -4.78893794e-02]
   [-3.58776616e-02 -2.09455938e-02 -6.46996414e-03 ... -8.20999889e-02
    -6.66456996e-02 -5.11782400e-02]]

  ...

  [[-1.33103990e-01 -1.22171898e-01 -1.11260477e-01 ... -1.65490900e-01
    -1.54812856e-01 -1.44001449e-01]
   [-1.42845919e-01 -1.31760003e-01 -1.20686634e-01 ... -1.75647136e-01
    -1.64839179e-01 -1.53889550e-01]
   [-1.52368788e-01 -1.41135424e-01 -1.29905858e-01 ... -1.85561953e-01
    -1.74630570e-01 -1.63550773e-01]
   ...
   [-1.02792132e-01 -9.23490470e-02 -8.19544077e-02 ... -1.33873502e-01
    -1.23604880e-01 -1.13228219e-01]
   [-1.13056942e-01 -1.02445049e-01 -9.18725171e-02 ... -1.44591236e-01
    -1.34180319e-01 -1.23652773e-01]
   [-1.23167280e-01 -1.12392354e-01 -1.01647534e-01 ... -1.55134554e-01
    -1.44588579e-01 -1.33916724e-01]]

  [[-1.33555458e-01 -1.22644696e-01 -1.11755622e-01 ... -1.65894371e-01
    -1.55228677e-01 -1.44433589e-01]
   [-1.43454318e-01 -1.32392451e-01 -1.21343555e-01 ... -1.76199810e-01
    -1.65406009e-01 -1.54475460e-01]
   [-1.53144294e-01 -1.41935867e-01 -1.30731662e-01 ... -1.86276997e-01
    -1.75363764e-01 -1.64303426e-01]
   ...
   [-1.02786708e-01 -9.23644383e-02 -8.19930797e-02 ... -1.33828880e-01
    -1.23568205e-01 -1.13205246e-01]
   [-1.13206888e-01 -1.02614761e-01 -9.20632185e-02 ... -1.44700608e-01
    -1.34298835e-01 -1.23785288e-01]
   [-1.23467928e-01 -1.12712550e-01 -1.01988456e-01 ... -1.55393254e-01
    -1.44857517e-01 -1.34200016e-01]]

  [[-1.33905659e-01 -1.23008104e-01 -1.12131789e-01 ... -1.66210510e-01
    -1.55555849e-01 -1.44771206e-01]
   [-1.43952635e-01 -1.32904992e-01 -1.21869925e-01 ... -1.76653597e-01
    -1.65875878e-01 -1.54960441e-01]
   [-1.53797613e-01 -1.42604085e-01 -1.31415519e-01 ... -1.86887051e-01
    -1.75981524e-01 -1.64939833e-01]
   ...
   [-1.02677609e-01 -9.22697985e-02 -8.19147469e-02 ... -1.33697228e-01
    -1.23437210e-01 -1.13082780e-01]
   [-1.13252799e-01 -1.02675211e-01 -9.21386376e-02 ... -1.44721102e-01
    -1.34322317e-01 -1.23818127e-01]
   [-1.23668633e-01 -1.12926915e-01 -1.02216238e-01 ... -1.55567441e-01
    -1.45036205e-01 -1.34388164e-01]]]

 [[[-4.30000192e-01 -4.26281803e-01 -4.22097729e-01 ... -4.37154580e-01
    -4.35430446e-01 -4.33087689e-01]
   [-4.13200226e-01 -4.09544387e-01 -4.05470382e-01 ... -4.20341934e-01
    -4.18606140e-01 -4.16263130e-01]
   [-3.96177486e-01 -3.92598971e-01 -3.88637267e-01 ... -4.03306327e-01
    -4.01564964e-01 -3.99212813e-01]
   ...
   [-4.78958728e-01 -4.75076465e-01 -4.70578639e-01 ... -4.86089588e-01
    -4.84430215e-01 -4.82072884e-01]
   [-4.62890971e-01 -4.59064755e-01 -4.54666929e-01 ... -4.70044082e-01
    -4.68358932e-01 -4.66004529e-01]
   [-4.46566500e-01 -4.42794109e-01 -4.38502849e-01 ... -4.53727395e-01
    -4.52020666e-01 -4.49672732e-01]]

  [[-4.27703209e-01 -4.24726828e-01 -4.21173149e-01 ... -4.32971529e-01
    -4.31817800e-01 -4.30066635e-01]
   [-4.11074014e-01 -4.08162037e-01 -4.04699246e-01 ... -4.16347930e-01
    -4.15161471e-01 -4.13418270e-01]
   [-3.94239825e-01 -3.91386681e-01 -3.88010001e-01 ... -3.99498511e-01
    -3.98285751e-01 -3.96555395e-01]
   ...
   [-4.76165059e-01 -4.73019605e-01 -4.69233379e-01 ... -4.81325863e-01
    -4.80308292e-01 -4.78585539e-01]
   [-4.60266901e-01 -4.57165888e-01 -4.53449906e-01 ... -4.65473861e-01
    -4.64405213e-01 -4.62665905e-01]
   [-4.44107333e-01 -4.41065582e-01 -4.37426006e-01 ... -4.49351791e-01
    -4.48237418e-01 -4.46486974e-01]]

  [[-4.24287974e-01 -4.21899792e-01 -4.18979702e-01 ... -4.27675587e-01
    -4.27154730e-01 -4.26025112e-01]
   [-4.07829168e-01 -4.05491527e-01 -4.02652366e-01 ... -4.11274184e-01
    -4.10702037e-01 -4.09553055e-01]
   [-3.91156298e-01 -3.88880472e-01 -3.86130291e-01 ... -3.94648552e-01
    -3.94030251e-01 -3.92869647e-01]
   ...
   [-4.72230911e-01 -4.69770806e-01 -4.66616808e-01 ... -4.75388785e-01
    -4.75039926e-01 -4.73988932e-01]
   [-4.56499987e-01 -4.54057336e-01 -4.50980068e-01 ... -4.59746174e-01
    -4.59337947e-01 -4.58257731e-01]
   [-4.40515843e-01 -4.38093699e-01 -4.35095376e-01 ... -4.43837588e-01
    -4.43371812e-01 -4.42265620e-01]]

  ...

  [[-6.16653025e-02 -7.04004585e-02 -7.89742293e-02 ... -3.52534168e-02
    -4.40180801e-02 -5.28459992e-02]
   [-5.24427636e-02 -6.10381323e-02 -6.94978131e-02 ... -2.65881355e-02
    -3.51460985e-02 -4.37875335e-02]
   [-4.31763898e-02 -5.16189639e-02 -5.99511926e-02 ... -1.79151782e-02
    -2.62549046e-02 -3.46977901e-02]
   ...
   [-8.90594983e-02 -9.81220509e-02 -1.06947681e-01 ... -6.12661370e-02
    -7.05519843e-02 -7.98423727e-02]
   [-7.99721111e-02 -8.89437462e-02 -9.77028338e-02 ... -5.25829515e-02
    -6.17136413e-02 -7.08688073e-02]
   [-7.08411721e-02 -7.97031798e-02 -8.83777906e-02 ... -4.39135849e-02
    -5.28701224e-02 -6.18708455e-02]]

  [[-4.39244092e-02 -5.27882487e-02 -6.15197992e-02 ... -1.73100131e-02
    -2.61114566e-02 -3.50067972e-02]
   [-3.49584065e-02 -4.36807817e-02 -5.22957826e-02 ... -8.89735094e-03
    -1.74947763e-02 -2.62050446e-02]
   [-2.59576267e-02 -3.45242394e-02 -4.30092473e-02 ... -4.95558750e-04
    -8.87404050e-03 -1.73836975e-02]
   ...
   [-7.06610379e-02 -7.98374754e-02 -8.88100550e-02 ... -4.27170027e-02
    -5.20231933e-02 -6.13629823e-02]
   [-6.17740215e-02 -7.08666413e-02 -7.97784816e-02 ... -3.42151214e-02
    -4.33712280e-02 -5.25823945e-02]
   [-5.28619087e-02 -6.18500875e-02 -7.06812727e-02 ... -2.57451721e-02
    -3.47336334e-02 -4.37970553e-02]]

  [[-2.62121696e-02 -3.51774526e-02 -4.40419383e-02 ...  5.17209794e-04
    -8.29205362e-03 -1.72251982e-02]
   [-1.74915942e-02 -2.63178174e-02 -3.50669599e-02 ...  8.69620789e-03
     8.67441858e-05 -8.66545607e-03]
   [-8.75583665e-03 -1.74254909e-02 -2.60426597e-02 ...  1.68369784e-02
     8.44431719e-03 -1.08710888e-04]
   ...
   [-5.23408840e-02 -6.15977461e-02 -7.06799583e-02 ... -2.43205830e-02
    -3.36258547e-02 -4.29905323e-02]
   [-4.36394597e-02 -5.28195222e-02 -6.18514920e-02 ... -1.59981092e-02
    -2.51542319e-02 -3.43920140e-02]
   [-3.49271951e-02 -4.40106009e-02 -5.29698890e-02 ... -7.71353523e-03
    -1.67049200e-02 -2.57998029e-02]]]

 [[[-1.27697068e-01 -1.39189479e-01 -1.50511973e-01 ... -9.24928489e-02
    -1.04322478e-01 -1.16007042e-01]
   [-1.30705156e-01 -1.42127137e-01 -1.53441611e-01 ... -9.55003676e-02
    -1.07309064e-01 -1.18962511e-01]
   [-1.33715910e-01 -1.45096280e-01 -1.56388105e-01 ... -9.85739871e-02
    -1.10349164e-01 -1.21984163e-01]
   ...
   [-1.19021630e-01 -1.30639966e-01 -1.41914474e-01 ... -8.39735697e-02
    -9.57932508e-02 -1.07476786e-01]
   [-1.21825562e-01 -1.33455016e-01 -1.44738196e-01 ... -8.67217345e-02
    -9.85556643e-02 -1.10251874e-01]
   [-1.24734041e-01 -1.36293420e-01 -1.47607981e-01 ... -8.95635563e-02
    -1.01401445e-01 -1.13099448e-01]]

  [[-1.17686445e-01 -1.29211661e-01 -1.40350036e-01 ... -8.28721424e-02
    -9.46352466e-02 -1.06242633e-01]
   [-1.20595811e-01 -1.32024192e-01 -1.43168632e-01 ... -8.58088352e-02
    -9.75465717e-02 -1.09126960e-01]
   [-1.23553810e-01 -1.34882197e-01 -1.46012821e-01 ... -8.88281161e-02
    -1.00528974e-01 -1.12068132e-01]
   ...
   [-1.09524055e-01 -1.20888378e-01 -1.32207838e-01 ... -7.46552154e-02
    -8.64265249e-02 -9.80450974e-02]
   [-1.12173580e-01 -1.23606228e-01 -1.34856180e-01 ... -7.72845865e-02
    -8.90632776e-02 -1.00690070e-01]
   [-1.14891866e-01 -1.26389982e-01 -1.37575298e-01 ... -8.00266670e-02
    -9.18026270e-02 -1.03424849e-01]]

  [[-1.07884923e-01 -1.19197363e-01 -1.30364431e-01 ... -7.32405378e-02
    -8.49568411e-02 -9.64993447e-02]
   [-1.10659585e-01 -1.22007145e-01 -1.33078913e-01 ... -7.61168796e-02
    -8.78012571e-02 -9.93128771e-02]
   [-1.13492847e-01 -1.24861729e-01 -1.35838141e-01 ... -7.90938141e-02
    -9.07345957e-02 -1.02204226e-01]
   ...
   [-1.00020109e-01 -1.11281059e-01 -1.22513648e-01 ... -6.53051982e-02
    -7.70457235e-02 -8.86110967e-02]
   [-1.02552091e-01 -1.13800572e-01 -1.25104643e-01 ... -6.78272256e-02
    -7.95714680e-02 -9.11394952e-02]
   [-1.05177389e-01 -1.16449498e-01 -1.27711479e-01 ... -7.04749230e-02
    -8.22114232e-02 -9.37721001e-02]]

  ...

  [[ 1.71415393e-01  1.61576733e-01  1.51605586e-01 ...  1.99845934e-01
     1.90577682e-01  1.81091911e-01]
   [ 1.72925690e-01  1.63107612e-01  1.53147219e-01 ...  2.01241774e-01
     1.92020687e-01  1.82572863e-01]
   [ 1.73908175e-01  1.64122254e-01  1.54179224e-01 ...  2.02047534e-01
     1.92897812e-01  1.83509223e-01]
   ...
   [ 1.63798130e-01  1.53937972e-01  1.43995133e-01 ...  1.92614318e-01
     1.83163289e-01  1.73548827e-01]
   [ 1.66817470e-01  1.56963985e-01  1.47007784e-01 ...  1.95489057e-01
     1.86106445e-01  1.76540551e-01]
   [ 1.69368038e-01  1.59520465e-01  1.49553270e-01 ...  1.97910015e-01
     1.88589669e-01  1.79066979e-01]]

  [[ 1.80262070e-01  1.70667897e-01  1.60929647e-01 ...  2.07923131e-01
     1.98914065e-01  1.89685813e-01]
   [ 1.82108299e-01  1.72533550e-01  1.62800509e-01 ...  2.09621879e-01
     2.00676428e-01  1.91497400e-01]
   [ 1.83395104e-01  1.73845497e-01  1.64127525e-01 ...  2.10755591e-01

This repeats 8 times, and each time the Momentum output has different numbers.

After the 8th time, we see this:

Momentum shape is:  (3, 32, 256, 256)
About to perform FFT
     4.09340482e-01  4.05844495e-01]]]]
Momentum shape is:  (3, 32, 256, 256)
About to perform FFT
     4.14556144e-01  4.19541179e-01]]]]
Momentum shape is:  (3, 32, 256, 256)
About to perform FFT
     0.15951451]]]]
Momentum shape is:  (3, 32, 256, 256)
About to perform FFT
     5.75028230e-01  5.80679185e-01]]]]
Momentum shape is:  (3, 32, 256, 256)
About to perform FFT
    -7.28528795e-02 -7.69183401e-02]]]]
Momentum shape is:  (3, 32, 256, 256)
About to perform FFT
    -1.11016810e-01 -1.22829347e-01]]]]
Momentum shape is:  (3, 32, 256, 256)
About to perform FFT
    0.14176391]]]]
Momentum shape is:  (3, 32, 256, 256)
About to perform FFT

So for some reason, it's printing things out of order. I'm assuming this has to do with the parallel programming?

aurora-cossairt commented 5 years ago

Latest issue

Traceback (most recent call last):
  File "/mnt/home/f0008575/src/energy-transfer-analysis/run_analysis.py", line 5, in <module>
    from FlowAnalysis import FlowAnalysis
  File "/mnt/home/f0008575/src/energy-transfer-analysis/FlowAnalysis.py", line 208
    self.FT_rho = self.FFT.forward(self.rho, self.FT_rho)
       ^
SyntaxError: invalid syntax

Interestingly, there is no complaint from line 191, in which I run self.FT_momentum[i] = self.FFT.forward(momentum[i], self.FT_momentum[i]) The only difference that I can see is that this line is run in a for loop. But I don't believe rho should need a for loop...?

aurora-cossairt commented 5 years ago

So, the previous issue was because I left a rogue tracer in the code... so it kept timing out while it waited for me to respond... oops.

Now we finally have real issues! When I run 

srun -n 1 --mem-per-cpu=20G --time=00:10:00 python ~/src/energy-transfer-analysis/run_analysis.py --res 256 --data_path
 /mnt/scratch/f0008575/a-1.00/id0/Turb.0010.vtk  --data_type Athena --type flow --eos isothermal -forced --outfile /mnt/scratch/f0008575/a-1.00/method_b_0010.hdf5 --ker
nel KernelGauss

I get the following error:

Traceback (most recent call last):
  File "/mnt/home/f0008575/src/energy-transfer-analysis/run_analysis.py", line 221, in <module>
    FA.run_analysis()
  File "/mnt/home/f0008575/src/energy-transfer-analysis/FlowAnalysis.py", line 238, in run_analysis
    self.momentum_filtered[j] = self.FFT.backward(FT_G * self.FT_momentum[j], self.momentum_filtered[j]) # We know FT_momentum is shape (3,256,256,256), so shape of self.FT_momentum[j] must be (256,256,256). Apparently something else (either momentum_filtered[j] or FT_G) is shape (223,)
ValueError: operands could not be broadcast together with shapes (223,) (256,256,256)

After investigating with some print statements, I discovered the problem: FT_G has shape (223, ) while self.momentum_filtered[j] has shape (256, 256, 256) so when I try to perform FT_G * self.momentum_filtered[j], it can't broadcast the shapes together. The reason FT_G is shape (223, ) is because it is based on k_bins, which is also shape (223, ). I use this in the kernel function where, before, we had k=Freq( ) # array of wavenumbers. self.momentum_filtered[j], on the other hand, is based on FT_momentum, which is based on momentum, both of which have shapes (256, 256, 256) because the resolution is 256. So, my guess is that either I should not be using k_bins in the kernel function, or I'm defining momentum incorrectly (and that problem is trickling down to self.momentum_filtered.

Also, I notice that when I create a Gaussian Kernel, only the first 43 entries are nonzero, and the remaining 180 entries are zeros:

Gaussian kernel is:  [9.02299856e-001 3.96422224e-001 7.65195145e-002 6.48923532e-003
 2.41780876e-004 3.95783310e-006 2.84642740e-008 8.99393722e-011
 1.24855218e-013 7.61501303e-017 2.04052401e-020 2.40225925e-024
 1.24252502e-028 2.82356324e-033 2.81901250e-038 1.23652696e-043
 2.38296291e-049 2.01761399e-055 7.50526422e-062 1.22659446e-068
 8.80730657e-076 2.77838443e-083 3.85077962e-091 2.34483398e-099
 6.27310619e-108 7.37327251e-117 3.80754493e-126 8.63847129e-136
 8.61064844e-146 3.77087318e-156 7.25529545e-167 6.13303415e-178
 2.27773277e-189 3.71652675e-201 2.66427384e-213 8.39126749e-226
 1.16113676e-238 7.05905054e-252 1.88545561e-265 2.21255177e-279
 1.14071637e-293 2.58386021e-308 2.47032823e-323 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000
 0.00000000e+000 0.00000000e+000 0.00000000e+000]

So that's weird.