Issue with reading data dated to 31 Mar from IAGA-2002 files (Mac OS)

[teraita]

When reading the data to the 1min values there is issue with handling of the data of 31 March. The file is not read correctly between 03-05UT. When manipulating the date information in filename, and data row (detestr and doy) in IAGA-2002 format to respond 31 Jan the file works. Might be some issue with epoch/ stream lengths?

Problem sees to hit only to period 03-05UT:

Same data file, date information manipulated:

When merging XYZ and F of the date and calculating deltaF the values are corrupted, so it this not only visualisation problem in MagPy.

Is this only Mac related issue?

Please, request data samples, if needed.

[leonro]

Hi Tero, can you please send me the data files? I will check what going on.

[teraita]

Hi Roman,

Here is the 1min SOD file with XYZ.

Cheers, Tero

Ps. We still have “chance” to get trough the group stage by stealing point from Belgium. At least about 3 hours, when Lukaku strikes… ;-)

On 21. Jun 2021, at 19.26, Roman Leonhardt @.***> wrote:

Hi Tero,

can you please send me the data files? I will check what going on. Sorry for not replying earlier, but my notebook broke down and I needed some time to set up everything again...

Best wishes, Roman

Dr. Roman Leonhardt Conrad Observatorium ZAMG - Zentralanstalt für Meteorologie und Geodynamik 1190 Wien, Hohe Warte 38 Tel.: +43 1 36026 2507 Fax: +43 1 36866 21 E-Mail: @.***https://webmail.zamg.ac.at/owa/redir.aspx?SURL=t9AS8jCcPggsx8oPf2D18nQbebRT-hDvZYGVU9uNLKVg80_NrjnSCG0AYQBpAGwAdABvADoAcgBvAG0AYQBuAC4AbABlAG8AbgBoAGEAcgBkAHQAQAB6AGEAbQBnAC4AYQBjAC4AYQB0AA..&URL=mailto%3aroman.leonhardt%40zamg.ac.at www.zamg.ac.athttps://webmail.zamg.ac.at/owa/redir.aspx?SURL=DgfuyOdHyWS5QXRRIyzOHmne5_jxCaCmQXd1H4B7AnJg80_NrjnSCGgAdAB0AHAAOgAvAC8AdwB3AHcALgB6AGEAbQBnAC4AYQBjAC4AYQB0AC8A&URL=http%3a%2f%2fwww.zamg.ac.at%2f

Conrad Observatorium 2763 Muggendorf , Trafelberg 1-2 Tel.: +43 2632 73276 oder 7367221 www.conrad-observatory.athttps://webmail.zamg.ac.at/owa/redir.aspx?SURL=3GEn2GkM89BDePjqOqAYT0d3pi9ZpUlcz8Zli53n9uNg80_NrjnSCGgAdAB0AHAAOgAvAC8AdwB3AHcALgBjAG8AbgByAGEAZAAtAG8AYgBzAGUAcgB2AGEAdABvAHIAeQAuAGEAdAAvAA..&URL=http%3a%2f%2fwww.conrad-observatory.at%2f

---

Von: Tero Raita @.***> Gesendet: Donnerstag, 20. Mai 2021 14:15:28 An: geomagpy/magpy Cc: Subscribed Betreff: [geomagpy/magpy] Issue with reading data dated to 31 Mar from IAGA-2002 files (Mac OS) (#103)

When reading the data to the 1min values there is issue with handling of the data of 31 March. The file is not read correctly between 03-05UT. When manipulating the date information in filename, and data row (detestr and doy) in IAGA-2002 format to respond 31 Jan the file works. Might be some issue with epoch/ stream lengths?

Problem sees to hit only to period 03-05UT: [SOD20190331_03-05UT]https://user-images.githubusercontent.com/22149196/118975443-b0e7da00-b97c-11eb-8cda-9aa3c9af40f9.png

[SOD20190331]https://user-images.githubusercontent.com/22149196/118975410-a9283580-b97c-11eb-8229-ddf25757a31c.png

Same data file, date information manipulated: [SOD20190131]https://user-images.githubusercontent.com/22149196/118975462-b6452480-b97c-11eb-883c-dcaa11e17398.png

When merging XYZ and F of the date and calculating deltaF the values are corrupted, so it this not only visualisation problem in MagPy.

Is this only Mac related issue?

Please, request data samples, if needed.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/geomagpy/magpy/issues/103, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD7YDSL7DIYWKM7LPESA643TOT4OBANCNFSM45G35ZEA.

EXTERNAL EMAIL: Do not click any links or open any attachments unless you trust the sender and know the content is safe.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/geomagpy/magpy/issues/103#issuecomment-865175015, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFI7QTDMLP2Y2WDTNMHSTNTTT5R3BANCNFSM45G35ZEA.

[teraita]

Hi,

2Hz IAGA-2002 files of 30-31Mar 2019.

Cheers, Tero

On 21. Jun 2021, at 19.32, Tero Raita @.***> wrote:

Hi Roman,

Here is the 1min SOD file with XYZ.

Cheers, Tero

Ps. We still have “chance” to get trough the group stage by stealing point from Belgium. At least about 3 hours, when Lukaku strikes… ;-)

> On 21. Jun 2021, at 19.26, Roman Leonhardt ***@***.*** ***@***.***>> wrote: > > > Hi Tero, > > > can you please send me the data files? I will check what going on. Sorry for not replying earlier, but my notebook broke down and I needed some time to set up everything again... > > > Best wishes, > Roman > > > Dr. Roman Leonhardt > Conrad Observatorium > ZAMG - Zentralanstalt für Meteorologie und Geodynamik > 1190 Wien, Hohe Warte 38 > Tel.: +43 1 36026 2507 > Fax: +43 1 36866 21 > E-Mail: ***@***.***> > www.zamg.ac.at > > > > Conrad Observatorium > 2763 Muggendorf , Trafelberg 1-2 > Tel.: +43 2632 73276 oder 7367221 > www.conrad-observatory.at > > ________________________________ > Von: Tero Raita ***@***.***> > Gesendet: Donnerstag, 20. Mai 2021 14:15:28 > An: geomagpy/magpy > Cc: Subscribed > Betreff: [geomagpy/magpy] Issue with reading data dated to 31 Mar from IAGA-2002 files (Mac OS) (#103) > > > When reading the data to the 1min values there is issue with handling of the data of 31 March. The file is not read correctly between 03-05UT. When manipulating the date information in filename, and data row (detestr and doy) in IAGA-2002 format to respond 31 Jan the file works. Might be some issue with epoch/ stream lengths? > > Problem sees to hit only to period 03-05UT: > [SOD20190331_03-05UT]> > > [SOD20190331]> > > Same data file, date information manipulated: > [SOD20190131]> > > When merging XYZ and F of the date and calculating deltaF the values are corrupted, so it this not only visualisation problem in MagPy. > > Is this only Mac related issue? > > Please, request data samples, if needed. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub>, or unsubscribe>. > > EXTERNAL EMAIL: Do not click any links or open any attachments unless you trust the sender and know the content is safe. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub , or unsubscribe . >

[leonro]

Well.... something I dont really understand: I am testing for leap seconds within the method LeapSeconds of stream.py. I am using the following conversion which actually should work fine in order to get a datetime compatible output:

t = "2019-03-31T02:00:00" ti = time.strptime(t, "%Y-%m-%dT%H:%M:%S") tim = datetime.fromtimestamp(time.mktime(ti))

tim, however, then results in 2019-03-31T03:00:00 which is obviously wrong. Wrong results are apparently only obtained on March 31, 2019 between 2:00 and 3:00.

Workaround: tim = datetime(*ti[:5]+(min(ti[5], 59),))

[leonro]

The workaround does the job. Issue solved in commit: e6e5258810c437b5d8399bc47338b0e884371f26