Data object not initializing N

[hayleyclev]

Coming across issue when attempting to initialize data object - could be due to the fact that the error field has not been specified

Error message:

C:\Users\hzibi\miniconda3\envs\lompe\Lib\site-packages\lompe\model\data.py:146: UserWarning: 'error' keyword not set for datatype 'convection'! Using error=50
  warnings.warn(f"'error' keyword not set for datatype '{datatype}'! Using error={error}", UserWarning)
C:\Users\hzibi\miniconda3\envs\lompe\Lib\site-packages\lompe\model\data.py:150: UserWarning: 'iweight' keyword not set for datatype 'convection'! Using iweight=1.0
  warnings.warn(f"'iweight' keyword not set for datatype '{datatype}'! Using iweight={iweight}", UserWarning)
Traceback (most recent call last):

  File ~\miniconda3\envs\lompe\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File c:\users\hzibi\documents\python scripts\untitled0.py:136
    pfisr_data = prepare_data(t - DT, t + DT)

  File c:\users\hzibi\documents\python scripts\untitled0.py:108 in prepare_data
    pfisr_data = lompe.Data(vlos, coordinates = coords, LOS = los, datatype = 'convection')

  File ~\miniconda3\envs\lompe\Lib\site-packages\lompe\model\data.py:187 in __init__
    self.error = np.full(self.N, error)

AttributeError: 'Data' object has no attribute 'N'

[Dartspacephysiker]

Hi Hayley,

I can see what may be going on. I ran the following from the command line (on a Linux machine), which is just a way of finding every file and line number where "self.N" occurs in the lompe directory:

find lompe/ -type f -iname '*.py' -print0 | xargs -0 grep -H -n "self.N"

I got back

./model/data.py:168:            self.N = self.values.shape[1] # number of data points
./model/data.py:170:            self.N = self.values.size
./model/data.py:187:            self.error = np.full(self.N, error)
./model/data.py:225:            self.N = self.values.shape[1]
./model/data.py:227:            self.N = self.values.size

Looking a little further into this, I see that on lines 167–170 of data.py, self.N only gets defined if the input data array has either one or two dimensions. Otherwise, self.N is left undefined.

We of course need to think about how to adapt this for data sets with more than two dimensions, but for now, can you reshape your array so that is either 1-D or 2-D? That should get rid of the error message.

[Dartspacephysiker]

Didn't mean to close this issue, clicked the wrong button :)

[jpreistad]

Hi Hayley, Me and Spencer discussed this issue this morning. As Spencer points out abobe, it seems like the error is due to unexpected input to the data class, namely that the data value variable (first argument when initializing the data class object) is not 1 or 2 dimensional. In this context, 1 or 2 dimensional is referring to the number of dimensions in the data value object, not if the data itself is of a 1,2, or 3D quantity. Each spatial dimension corresponds to a row in the data value variable. So for 3D magnetometer data, the array would be 2D but have 3 rows, and as many columns as data points.

My suspicion is that your data value object may be something different, having e.g. 3 dimensions. Hopefully some of the notebook examples will be helpful in putting these arrays together. Here is an example of how 3D magnetometer data are stacked together: smag_B = np.vstack((smag.Be.values, smag.Bn.values, smag.Bu.values)) # nT

We further discussed how to best approach the 3D nature of the LOS data from AMISR into Lompe. Since Lompe by design only treat the horizontal convection, we suggest that the processing of AMISR data that is needed to arrive at the projected horizontal LOS velocities that will go into lompe is made outside the data object. However, we are very interested in this step ourselves, and are happy to discuss possible strategies, and of-course include code that do this pre-processing step. This code could e.g. fit into our ´./lompe/data_tools/dataloader.py´ set of functions that pre-process data for use in Lompe.

[hayleyclev]

Jone and Spencer,

Thank you both so much for looking into this so quickly! Both Leslie Lamarche and myself (added to the thread, I FWD’d to her today) have been working with the library to process PFISR data and have gone through a series of reshaping processes for the input data. This is very helpful information - we were able to find the data.py area in which this was occurring and were able to reshape our data accordingly.

I apologize for the delayed response - and thanks again to you both for getting back to me so quickly! I will let you know if we come to a good and we’ll documentable solution to close it out soon enough.

-Hayley Clevenger

---

From: Jone Peter Reistad @.> Sent: Friday, September 8, 2023 7:52:22 AM To: klaundal/lompe @.> Cc: Clevenger, Hayley L. @.>; Author @.> Subject: [EXTERNAL] Re: [klaundal/lompe] Data object not initializing N (Issue #26)

CAUTION: This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Hayley, Me and Spencer discussed this issue this morning. As Spencer points out abobe, it seems like the error is due to unexpected input to the data class, namely that the data value variable (first argument when initializing the data class object) is not 1 or 2 dimensional. In this context, 1 or 2 dimensional is referring to the number of dimensions in the data value object, not if the data itself is of a 1,2, or 3D quantity. Each spatial dimension corresponds to a row in the data value variable. So for 3D magnetometer data, the array would be 2D but have 3 rows, and as many columns as data points.

My suspicion is that your data value object may be something different, having e.g. 3 dimensions. Hopefully some of the notebook examples will be helpful in putting these arrays together. Here is an example of how 3D magnetometer data are stacked together: smag_B = np.vstack((smag.Be.values, smag.Bn.values, smag.Bu.values)) # nT

We further discussed how to best approach the 3D nature of the LOS data from AMISR into Lompe. Since Lompe by design only treat the horizontal convection, we suggest that the processing of AMISR data that is needed to arrive at the projected horizontal LOS velocities that will go into lompe is made outside the data object. However, we are very interested in this step ourselves, and are happy to discuss possible strategies, and of-course include code that do this pre-processing step. This code could e.g. fit into our ´./lompe/data_tools/dataloader.py´ set of functions that pre-process data for use in Lompe.

— Reply to this email directly, view it on GitHubhttps://github.com/klaundal/lompe/issues/26#issuecomment-1711549447, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3GR2RBQWKAQOORT37JOWR3XZMBHNANCNFSM6AAAAAA4PDX3L4. You are receiving this because you authored the thread.Message ID: @.***>