Error with .radars.sqlite

[ReedOnly]

Hi,

I believe there is an error with the .radars.sqlite database as my script is not able to fetch the site object for plotting scans.

When running this code:

import datetime
from davitpy import pydarn

site = pydarn.radar.site(code='cly', dt=datetime.datetime(2014,12,15,15,0))

I get this resulting error:

  File "/usr/local/lib/python2.7/dist-packages/davitpy-0.5-py2.7-linux-x86_64.egg/davitpy/pydarn/radar/radStruct.py", line 763, in __init__
    self.fillFromSqlite(dbname, radId, dt=dt)

  File "/usr/local/lib/python2.7/dist-packages/davitpy-0.5-py2.7-linux-x86_64.egg/davitpy/pydarn/radar/radStruct.py", line 809, in fillFromSqlite
    self.id = row[0]

TypeError: 'NoneType' object has no attribute '__getitem__

When checking the .radars.sqlite database it seems like the radId values are wrong, there is 11 radars with id 33 and none with 66 which is why python is returning NoneType for the site object for the cly radar it's trying to look up.

Can anyone delete and update their .radars.sqlite file in the /tmp/sd folder and see if you get the same error?

Might this be related to the activation of the Svalbard svb radar last week?

Regards, Kristian Reed

[ksterne]

Yes, believe there was an error in updating the hdw.dat database.

Working on it...

[asreimer]

I can confirm that this is a problem. In the limited testing that I've just done, I can't get this to work for any radar.

[ksterne]

Looks the issue is with the hdwRead function not working properly. Looks like @aburrell introduced a problem back with 30431406a4a1da1e4118ade8682cd762f625075a during the big merge conflict with the fall 2015 release. Funny as she was one of the first to point out there was an issue here! ha.

In particular the if statement was changed up so that if the year isn't 2999, then no values are written: https://github.com/vtsuperdarn/davitpy/blob/develop/davitpy/pydarn/radar/radInfoIo.py#L191-L209

Soooo, any hdw.dat file that doesn't have more than one line returns only the tval variable filled. I was monkeying around with our update script here and thought this tval variable was something else which is why some hdw.dat lines weren't being written to the database.

I'll have a subsequent pull request or I can even hotfix it if appropriate. @aburrell, you can cherry-pick the commit for your demo if you need.

[ksterne]

Believe I've got the hdw.dat database updated now. Can @ReedOnly, @asreimer, @aburrell confirm?

[aburrell]

Good news: everything works for me now.

@ksterne don't give me too much credit, I didn't introduce the 2999 line. I should have spoken up about the possibilities for problems, of course. We can fix that by using today's date as the comparison.

That said, I don't see the part where if the year isn't 2999 no values are written? Looking at the code it goes straight to writing values if the year isn't 2999. I'm happy to fix the 2999 hardcode as an upper limit, but I don't see the other issue you're talking about so I can't fix it.

Current code snippet:

    if len(ldat) == 0:
        continue
    if ldat[0] == '#':
        continue
    if int(ldat[1]) == 2999:   #<-------propose changing this to dt.datetime.today().year
        siteF['tval'].append(-1) # If year is 2999 no data is written
    else:
        # If year is not 2999, write data
        siteF['tval'].append(timeYrsecToDate(int(ldat[2]), int(ldat[1])))
        siteF['geolat'].append(float(ldat[3]))
        siteF['geolon'].append(float(ldat[4]))
        siteF['alt'].append(float(ldat[5]))
        siteF['boresite'].append(float(ldat[6]))
        siteF['bmsep'].append(float(ldat[7]))
        siteF['vdir'].append(float(ldat[8]))
        siteF['atten'].append(float(ldat[9]))
        siteF['tdiff'].append(float(ldat[10]))
        siteF['phidiff'].append(float(ldat[11]))
        siteF['interfer'].append([float(ldat[12]), float(ldat[13]),
                                  float(ldat[14])])
        siteF['recrise'].append(float(ldat[15]))
        siteF['maxatten'].append(int(ldat[16]))
        siteF['maxgate'].append(int(ldat[17]))
        siteF['maxbeam'].append(int(ldat[18]))

Also, we should also fix the error reporting on this routine. It doesn't inform the user about any of the problems it encounters, including incorrect input.

[ksterne]

Hey @aburrell,

The 2999 value comes from the hdw.dat convention that the second value of the line is the year until the line is good. Since I guess we don't forsee SuperDARN lasting until the year 3000, that value was chosen to represent the present. Here the convention has changed some to where the value is represented by -1. Changing the value from 2999 is not recommended as it breaks things in RST code since assumptions have been made there.

Setting of this tval wasn't the problem with the database update script though. The problem occurs for the later lines that set geolat, geolon, alt, and so on. You see, if the "present" line is encountered (year == 2999), then none of these values are written. Basically, these lines were indented and they didn't need to be.

But despite all of this, I'm glad things got working now!

Any thoughts from you or others on hotfixing this versus pull request? If there's no opinion by the end of the eastern US day, I'll make a pull request for it.

[aburrell]

Hi @ksterne ,

Alright, I see where I went wrong and why (thanks for all the off-thread discussion). Now that we've looked at this code more, here are the things I think should be fixed:

1) Error handling (for example, checking input and examining results from sql calls) is missing. I've opened Issue #279 (for after the next release)

2) Comparing to year 2999 (a millennium bug in the waiting). I've opened Issue #280. (not high priority, even after release).

3) The tabbing issue Kevin found, which is a bug since we always want to return at least the last entry from any radar (even if that radar isn't currently operational). (High priority).

[aburrell]

Hi Kristian,

I’ve encountered the same error and Kevin is working on it. It’s “good" to see more people are experiencing the same problems.

Do you have an old .radars.sqlite database you can use until the problem is fixed?

Cheers, Angeline

On 26 Oct 2016, at 01:07, ReedOnly notifications@github.com wrote:

Hi,

I Believe there is an error with the .radars.sqlite database as my script is not able to fetch the site object for plotting scans.

When running this code: `import datetime from davitpy import pydarn

t=datetime.datetime(2014,12,15,15,0) site = pydarn.radar.site(code='cly', dt=datetime.datetime(2014,12,15,15,0))`

I get this resulting error: ` File "/home/kristian/Documents/rfefinder_velMatrix/test2.py", line 5, in site = pydarn.radar.site(code='cly', dt=datetime.datetime(2014,12,15,15,0))

File "/usr/local/lib/python2.7/dist-packages/davitpy-0.5-py2.7-linux-x86_64.egg/davitpy/pydarn/radar/radStruct.py", line 763, in init self.fillFromSqlite(dbname, radId, dt=dt)

File "/usr/local/lib/python2.7/dist-packages/davitpy-0.5-py2.7-linux-x86_64.egg/davitpy/pydarn/radar/radStruct.py", line 809, in fillFromSqlite self.id = row[0]

TypeError: 'NoneType' object has no attribute 'getitem'`

When checking the .radars.sqlite database it seems like the radId values are wrong, there is 11 radars with id 33 and none with 66 which is why python is returning NoneType for the site object for the cly radar it's trying to look up.

Can anyone delete and update their .radars.sqlite file in the /tmp/sd folder and see if you get the same error?

Might this be related to the activation of the Svalbard svb radar last week?

Regards, Kristian Reed

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/277, or mute the thread https://github.com/notifications/unsubscribe-auth/AGuC_u7A07M0EthoRv8J4xKputGW_043ks5q3pnfgaJpZM4KgntN.

[ksterne]

Back to my comment a little bit ago, this problem should be fixed and you can re-run the code you've posted here @ReedOnly. Mostly not closing this issue for wanting comment on hotfixing develop or pull request and testing on issue related to this, but not directly. I think @aburrell seemed in favor of pull request for this so I'll work on that this afternoon.

@ReedOnly, have you been able to run the radar.site command? If not, then I'd recommend running radar.updateRadars() which will force an update to the sqlite file.

[asreimer]

A hotfix for this is appropriate.

[ReedOnly]

Everything seems to work for me again now. Thanks @ksterne and the rest of you for getting around this bug so quickly!

Cheers, Kristian

[ksterne]

Great to hear it. I've applied the hotfix for the underlying issue here (not that the hdw.dat sql databse was broken) with a00c429 directly to develop. Thanks for comments and finding ways we can better setup the hdwRead function though team. Closing now...