search

vtsuperdarn / davitpy

DEPRECATED The DaViT Python project
http://vtsuperdarn.github.com/davitpy/
GNU General Public License v3.0
37 stars 59 forks source link

Davitpy looking for hdw.dat.svb #333

Closed kkotyk closed 6 years ago

kkotyk commented 6 years ago 
DEBUG:root:Trying to connect to hdw.dat mongodb
ERROR:root:Could not connect to remote DB: mongodb://sd_dbread:5d@sd-work9.ece.vt.edu:27017/radarInfo
WARNING:root:Reading hdw.dat info from local files in /home/kotyk/Documents/rst/tables/superdarn/hdw/
ERROR:root:hdwRead: cannot read /home/kotyk/Documents/rst/tables/superdarn/hdw/hdw.dat.svb
Traceback (most recent call last):
  File "/home/kotyk/davitpyenv/lib/python2.7/site-packages/davitpy-0.8-py2.7-linux-x86_64.egg/davitpy/pydarn/radar/radInfoIo.py", line 156, in hdwRead
    file_hdw = open(pathOpen, 'r')
IOError: [Errno 2] No such file or directory: '/home/kotyk/Documents/rst/tables/superdarn/hdw/hdw.dat.svb'
You may be getting this error because your computer cannot contact an appropriate internet server to get the latest hdw.dat information.  You can can use a local file instead by setting the SD_HDWPATH environment variable to the location of the local hdw.dat path.

First of all, cant connect to that DB. Not sure if thats an issue. But the other issue here is that something in Davitpy is looking for hdw.dat.svb. I seem to recall that this was removed or renamed maybe? @asreimer since you are my goto. I get this issue when loading up davitpy the first time after a fresh install.

asreimer commented 6 years ago

You probably can't connect to the DB because the port is being blocked on your end. When I visited usask in December, I noticed they have really locked things down now.

The missing hdw.dat.svb is not a bug with DaViT-py. This is actually an RST problem.

See here: https://github.com/vtsuperdarn/hdw.dat/issues/5 and https://github.com/SuperDARN/rst/issues/114

We do however need to provide better instructions on how to the hdw.dat.* files instead. In fact we need to tear out the mongo stuff completely. I have a rocket campaign I'm supporting at work and a PhD defense on Feb 6. After that I can finally return to fixing this project.

In the mean time, update your radars.dat file.

ksterne commented 6 years ago

So, root cause seems to be that mongodb wasn't running on our sd-work9.ece.vt.edu server. Weird. I started it up manually and just ran an update for files to the database manually. Seems as though things are working now as I saw this same problem working on a pull request and now that test is working. @kkotyk, can you confirm this is working for you?

Sorry for taking a while to get to this!!! I agree with @asreimer on needing to at least revisit the usefulness of a mongodb database for these files. I think it was originally a good idea when we thought we'd be able to take other data products to mongodb or some other kind of database to allow for faster run times.

kkotyk commented 6 years ago

I just tried again and I still cannot connect to remote DB when starting davitpy. Still just defaulting to local files.

salsrag commented 6 years ago

Kevin,

I do not know if you have solved this problem or it will solve your problem, bur I recently installed davitpy on my machine and had a similar problem. I assume that you will be starting davitpy from a terminal window on which a lot of messages are written. One of the last messages is for a token that written into you browser. This token must be copied into the address window of the browser that will be used to display the notebook. If you do this and repeat the “jupyter notebook” command, it should work.

Good luck, Ray

From: kkotyk [mailto:notifications@github.com] Sent: Monday, February 12, 2018 12:23 PM To: vtsuperdarn/davitpy davitpy@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [vtsuperdarn/davitpy] Davitpy looking for hdw.dat.svb (#333)

I just tried again and I still cannot connect to remote DB when starting davitpy. Still just defaulting to local files.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/333#issuecomment-364996199 , or mute the thread https://github.com/notifications/unsubscribe-auth/AIj04sS810rH3h3b7Evh5t5pBSNYpbCiks5tUHNegaJpZM4Rr6iP .

ksterne commented 6 years ago

@kkotyk, I'm guessing this might be related to what @asreimer was noting with network blockages. Have you been able to do a netcat or traceroute to sd-work9.ece.vt.edu to see if you're able to see port 27017? I think I've cleared up the problem here, but without anyone else testing you're my only datapoint that's outside the VT network.

Ray, I'm not sure if that was directed to me or to Keith, but this sounds different than the issue here which was not being able to reach the sd-work9.ece.vt.edu hdw.dat database. That sounds more like an issue with running jupyter from what I can gather.

abhinavralhan commented 6 years ago

A similar issue that I am facing:

ERROR:root:Could not connect to remote DB: mongodb://sd_dbread:5d@sd-work9.ece.vt.edu:27017/radarInfo WARNING:root:Reading hdw.dat info from local files in None ERROR:root:radarRead: cannot read None Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/davitpy-0.8-py2.7-linux-x86_64.egg/davitpy/pydarn/radar/radInfoIo.py", line 67, in radarRead file_net = open(pathOpen, 'r') TypeError: coercing to Unicode: need string or buffer, NoneType found ERROR:root:Could not update .radars.sqlite file with hdw.dat info

And another one, since I guess this is a similar issue, I'll post this too:

ERROR:root:/tmp/sd/.radars.sqlite not found

aburrell commented 6 years ago

So I encountered this error as well. I did some googling and I believe it's a problem with a matplotlib update. Once I updated my python modules the error went away.

snehasinghania commented 6 years ago

@ksterne I'm able to replicate this issue with the error: ERROR:root:Could not connect to remote DB: mongodb://sd_dbread:5d@sd-work9.ece.vt.edu:27017/radarInfo

Surprisingly, I have access to port 27017 to sd-work9.ece.vt.edu, verified using netcat: Connection to sd-work9.ece.vt.edu 27017 port [tcp/*] succeeded!

Therefore I tried importing davitpy a second time (by chance) in Python and it succeeded without any error. I'm able to establish a data pipeline with the VT server and download data.

This error occurs only during the first import of davitpy in Python. All subsequent imports work fine. I suspect there is an issue during initialization somewhere. Hope this helps. Will also investigate further.

snehasinghania commented 6 years ago

@abhinavralhan You are facing two issues, a DB connection issue and not being able to update the .radars.sqlite file. Since I have faced both these issues, I have outlined how I resolved them.

A hotfix for your first issue, to connect to the DB, would be to import davitpy twice as outlined above.

For the second issue, either update the .radars.sqlite file by obj = davitpy.pydarn.radar.updateRadars(). If this doesn't work, try:

For more check out #339 and #300. Hope this helps.