irfu / irfu-matlab

Matlab routines to work with space data, particularly with MMS and Cluster/CAA data. Also some general plasma routines.
59 stars 46 forks source link

How to access mms data base using functions given in irf-matlab devel. #69

Closed neetasha closed 3 years ago

neetasha commented 3 years ago

Step 1: Latest code?

Step 2: Describe your environment

Step 3: Describe the problem

Hello sir/mam, I am new to the field of space data analysis. I am using irfu-matlab devel package of GitHub for data analysis of MMS spacecraft. Initially I started with few examples (\plots\examples) which access data from CAA. Later I started with MMS examples (\plots\mms) given in the irfu-matlab devel. There I am stuck with the function cval.m I downloaded the data form cdaweb for 2016-11-28 from MMS spacecraft and stored that data into the current folder irf-devel (irfu-devel\data). After evaluating mms.init(), I started using examples where cval.m is showing no data base present. Sometimes it gives the result matrix say B=[], but with 0X0 matrix i.e. data has not been read by it.

Will you Please try to guide me with this issue?

Expected behavior The function ,"c_eval('Bxyz1=mms.db_get_ts(''mms1_fgms_brst_l2_20161128072930_20161128073119'',''mms1_fgm_b_gse_brst_l2'',Tint);');" should show the values of B1 which are magnetic field data taken from cdaweb website. The first entry inside the () is file name, the second entry is variable name of the file while the third entry if Time, which is Tint = irf.tint mms1_fgms_brst_l2_20161128072930_20161128073119.zip

('2016-11-28T07:29:00.00Z/2016-11-28T07:31:30.00Z');

Actual behavior The function when used shows B1 with 0X0 matrix i.e. no data find. The data file which I am using is attached below. Please do refer it.

mms1_fgms_brst_l2_20161128072930_20161128073119.zip

thomas-nilsson-irfu commented 3 years ago

Dear @neetasha, one thing which we have perhaps not explained that well for new users is that the code in the mms.db_* functions will look for data in subfolders to the path provided with mms.db_init and that the subfolder structure should be similar to that of MMS SDC (see for instance https://lasp.colorado.edu/mms/sdc/public/data/mms1/fgm/brst/l2/2016/11/28/). That is to say specifically for burst data only it is to be sorted in one more level (daily subfolder) compared with the NASA cdaweb which put all files in subfolder down to month only (see https://cdaweb.gsfc.nasa.gov/pub/data/mms/mms1/fgm/brst/l2/2016/11/). Monthly folders is however the lowest level subfolder for survey (fast/slow) data at MMS SDC and NASA cdaweb.

Did you create such a folder structure in your local downloaded root-path which you used with mms.db_inti()? I.e. for mms.db_init('local_file_db','c:\Whatever\Your\Path\'), do you have subfolders mms1, mms2, mms3, mms4 in this example path c:\Whatever\Your\Path\, and in these subfolder do you have next level subfolders fgm and so on down to daily folders for burst data?

I think there might be a typo in your command, fgms should be simply fgm as per the file names at MMS SDC and NASA cdaweb (see for instance: https://lasp.colorado.edu/mms/sdc/public/data/mms1/fgm/brst/l2/2016/11/28/).

And just minor question: are you running with a 64 bit installation? (There is a checkbox A 64-bit installation of MATLAB was used. in our bug template which you left unchecked / did not put an X inside of). The reason we have this checkbox and that I'm asking now is that some of the code specifically used to read the cdf files is dependent upon compiled mex files which are only tested on our end for 64-bit installations. If you run on a 32bit version it could be that these fail and are not able to read the cdf files and return any data from your download into Matlab.

neetasha commented 3 years ago

Hello Sir, Yes, I am running with a 64 bit installation of MATLAB and I am extremely sorry for the delay in response. Firstly, Thanks a lot sir for helping me out with this issue. The above asked issue is solved! Secondly, The program which I am trying to run was one of the example of (plots\mms\Example_MMS_B_E_J) , when I evaluate these lines "R = mms.get_data('R_gse',Tint); c_eval('Rxyz? = irf.ts_vec_xyz(R.time,R.gseR?);');" then error occurs: "Reference to non-existent field 'gseR1'. Error in c_eval (line 48) for cl_id=sc_list_1, evalin('caller', irf_ssub(ev_str, cl_id)), end"

I have downloaded the data, attached below, from SDC-MMS. Which data should I refer in order to resolve the error? I am unable to understand the var_name ('R_gse'), because such variable name I could not find in the website SDC-MMS. Either the var_name is in this form (mms1_mec_r_gse) or other. Would you please like to help me out!

Thanks and regards, Neetasha Arya.

On Mon, Mar 8, 2021 at 7:31 AM Thomas Nilsson @.***> wrote:

Dear @neetasha https://github.com/neetasha, one thing which we have perhaps not explained that well for new users is that the code in the mms.db_* functions will look for data in subfolders to the path provided with mms.db_init and that the subfolder structure should be similar to that of MMS SDC (see for instance https://lasp.colorado.edu/mms/sdc/public/data/mms1/fgm/brst/l2/2016/11/28/). That is to say specifically for burst data only it is to be sorted in one more level (daily subfolder) compared with the NASA cdaweb which put all files in subfolder down to month only (see https://cdaweb.gsfc.nasa.gov/pub/data/mms/mms1/fgm/brst/l2/2016/11/). Monthly folders is however the lowest level subfolder for survey (fast/slow) data at MMS SDC and NASA cdaweb.

Did you create such a folder structure in your local downloaded root-path which you used with mms.db_inti()? I.e. for mms.db_init('local_file_db','c:\Whatever\Your\Path\'), do you have subfolders mms1, mms2, mms3, mms4 in this example path c:\Whatever\Your\Path\, and in these subfolder do you have next level subfolders fgm and so on down to daily folders for burst data?

I think there might be a typo in your command, fgms should be simply fgm as per the file names at MMS SDC and NASA cdaweb (see for instance: https://lasp.colorado.edu/mms/sdc/public/data/mms1/fgm/brst/l2/2016/11/28/ ).

And just minor question: are you running with a 64 bit installation? (There is a checkbox A 64-bit installation of MATLAB was used. in our bug template which you left unchecked / did not put an X inside of). The reason we have this checkbox and that I'm asking now is that some of the code specifically used to read the cdf files is dependent upon compiled mex files which are only tested on our end for 64-bit installations. If you run on a 32bit version it could be that these fail and are not able to read the cdf files and return any data from your download into Matlab.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/irfu/irfu-matlab/issues/69#issuecomment-792833703, or unsubscribe https://github.com/notifications/unsubscribe-auth/AS3MBFEIX2KJUJ2FS6KFZHDTCTUU7ANCNFSM4YYYG7KA .

thomas-nilsson-irfu commented 3 years ago

Dear @neetasha, glad to hear the original issue was resolved.

As for the issue with R = mms.get_data('R_gse',Tint);, I will try to explain a bit.

Here the name R_gse is used as a bit of shorthand for position in GSE coordinates, and details are found in mms.get_data, more specifically the following lines: https://github.com/irfu/irfu-matlab/blob/c7204744a027f619a5c5f55327bc3ee9158f6a85/mission/mms/%2Bmms/get_data.m#L277-L300

As can be seen in there, the code tries to load ancillary information such as the computed position from a number of different source files, starting with the most accurate source and having a few fallback options with somewhat lower data-level products (if the preferred higher data-level source file is not yet available for the requested interval). This way the user can simply request R_gse and get the best available position (or velocity if V_gse is requested).

The files you would need to have in your copy is (in decreasing preferential order):

The MEC L2 files are typically released after about 1-2 months, while the DFG QL are available for the instrument teams after just a few hours (but with less calibrations & accuracy). As the time interval you were interested in is way back it 2016, I would think the MEC L2 files should cover it perfectly well.

So to make a long story short, you would need to have the mms[1-4]_mec_srvy_l2_epht89d files available in your data folder (in corresponding subfolders as on SDC).

With best regards,

Thomas Nilsson

neetasha commented 3 years ago

The issue is resolved. Now, the Example is working smoothly. Thanks a lot sir.

On Mon, Mar 29, 2021 at 12:50 AM Thomas Nilsson @.***> wrote:

Dear @neetasha https://github.com/neetasha, glad to hear the original issue was resolved.

As for the issue with R = mms.get_data('R_gse',Tint);, I will try to explain a bit.

Here the name R_gse is used as a bit of shorthand for position in GSE coordinates, and details are found in mms.get_data, more specifically the following lines:

https://github.com/irfu/irfu-matlab/blob/c7204744a027f619a5c5f55327bc3ee9158f6a85/mission/mms/%2Bmms/get_data.m#L277-L300

As can be seen in there, the code tries to load ancillary information such as the computed position from a number of different source files, starting with the most accurate source and having a few fallback options with somewhat lower data-level products (if the preferred higher data-level source file is not yet available for the requested interval). This way the user can simply request R_gse and get the best available position (or velocity if V_gse is requested).

The files you would need to have in your copy is (in decreasing preferential order):

  • mms[1-4]_mec_srvy_l2_epht89d (preferred)
  • mms[1-4]_dfg_srvy_l2pre
  • mms[1-4]_fgm_srvy_l2
  • mms[1-4]_dfg_srvy_ql (last fallback)

The MEC L2 files are typically released after about 1-2 months, while the DFG QL are available for the instrument teams after just a few hours (but with less calibrations & accuracy). As the time interval you were interested in is way back it 2016, I would think the MEC L2 files should cover it perfectly well.

So to make a long story short, you would need to have the mms[1-4]_mec_srvy_l2_epht89d files available in your data folder (in corresponding subfolders as on SDC https://lasp.colorado.edu/mms/sdc/public/data/mms1/mec/srvy/l2/epht89d/2016/11/ ).

With best regards,

Thomas Nilsson

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/irfu/irfu-matlab/issues/69#issuecomment-809156014, or unsubscribe https://github.com/notifications/unsubscribe-auth/AS3MBFCIJ7GE3YFUOZKA7ATTGA5OXANCNFSM4YYYG7KA .

thomas-nilsson-irfu commented 3 years ago

Dear @neetasha, glad to hear it all worked out!

(I'm closing this issue here, if you experience any new issues feel free to create a new issue on github).