Tsyganenko tracing not working correctly

[johncoxon]

I've just installed davitpy on my Mac and I've had a couple of issues. Firstly, when I initially tried to import the module, it warned me that pyproj was not installed, so I had to pip install pyproj to fix this issue.

Secondly, when I run the example code given in the source of the submodule:

lats = np.arange(10,90,10)
lons = np.zeros(len(lats))
rhos = 6372. * np.ones(len(lats))
trace = tsy.tsygTrace(lats, lons, rhos)

I get the following:

/Users/jc3e14/anaconda/lib/python2.7/site-packages/davitpy-0.7-py2.7-macosx-10.7-x86_64.egg/davitpy/models/tsyganenko/__init__.py:131: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  assert (None not in [lat, lon, rho]) or filename, 'You must provide either (lat, lon, rho) or a filename to read from'
/Users/jc3e14/anaconda/lib/python2.7/site-packages/davitpy-0.7-py2.7-macosx-10.7-x86_64.egg/davitpy/models/tsyganenko/__init__.py:133: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  if None not in [lat, lon, rho]:

  RECALC_08: RADIAL SOLAR WIND --> GSW SYSTEM IDENTICAL HERE
  TO STANDARD GSM (I.E., XGSW AXIS COINCIDES WITH EARTH-SUN LINE)

 **** RECALC_08 WARNS: YEAR IS OUT OF INTERVAL 1965-2015: IYEAR=2017
      CALCULATIONS WILL BE DONE FOR IYEAR=2015

/Users/jc3e14/anaconda/lib/python2.7/site-packages/davitpy-0.7-py2.7-macosx-10.7-x86_64.egg/davitpy/models/tsyganenko/__init__.py:355: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  self.xTrace[ip,self.l[ip]:self.l[ip]+l] = xarr[0:l]
/Users/jc3e14/anaconda/lib/python2.7/site-packages/davitpy-0.7-py2.7-macosx-10.7-x86_64.egg/davitpy/models/tsyganenko/__init__.py:356: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  self.yTrace[ip,self.l[ip]:self.l[ip]+l] = yarr[0:l]
/Users/jc3e14/anaconda/lib/python2.7/site-packages/davitpy-0.7-py2.7-macosx-10.7-x86_64.egg/davitpy/models/tsyganenko/__init__.py:357: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  self.zTrace[ip,self.l[ip]:self.l[ip]+l] = zarr[0:l]

 **** RECALC_08 WARNS: YEAR IS OUT OF INTERVAL 1965-2015: IYEAR=2017
      CALCULATIONS WILL BE DONE FOR IYEAR=2015

 **** RECALC_08 WARNS: YEAR IS OUT OF INTERVAL 1965-2015: IYEAR=2017
      CALCULATIONS WILL BE DONE FOR IYEAR=2015

 **** RECALC_08 WARNS: YEAR IS OUT OF INTERVAL 1965-2015: IYEAR=2017
      CALCULATIONS WILL BE DONE FOR IYEAR=2015

 **** RECALC_08 WARNS: YEAR IS OUT OF INTERVAL 1965-2015: IYEAR=2017
      CALCULATIONS WILL BE DONE FOR IYEAR=2015

 **** RECALC_08 WARNS: YEAR IS OUT OF INTERVAL 1965-2015: IYEAR=2017
      CALCULATIONS WILL BE DONE FOR IYEAR=2015

 **** RECALC_08 WARNS: YEAR IS OUT OF INTERVAL 1965-2015: IYEAR=2017
      CALCULATIONS WILL BE DONE FOR IYEAR=2015

 **** RECALC_08 WARNS: YEAR IS OUT OF INTERVAL 1965-2015: IYEAR=2017
      CALCULATIONS WILL BE DONE FOR IYEAR=2015

 **** COMPUTATIONS IN THE SUBROUTINE TRACE_08 ARE TERMINATED: THE NUMBER OF POINTS EXCEEDED LMAX ****

/Users/jc3e14/anaconda/lib/python2.7/site-packages/davitpy-0.7-py2.7-macosx-10.7-x86_64.egg/davitpy/models/tsyganenko/__init__.py:361: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  self.xTrace = self.xTrace[:,0:self.l.max()]
/Users/jc3e14/anaconda/lib/python2.7/site-packages/davitpy-0.7-py2.7-macosx-10.7-x86_64.egg/davitpy/models/tsyganenko/__init__.py:362: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  self.yTrace = self.yTrace[:,0:self.l.max()]
/Users/jc3e14/anaconda/lib/python2.7/site-packages/davitpy-0.7-py2.7-macosx-10.7-x86_64.egg/davitpy/models/tsyganenko/__init__.py:363: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  self.zTrace = self.zTrace[:,0:self.l.max()]

When I run trace.plot() I get the following:

/Users/jc3e14/anaconda/lib/python2.7/site-packages/davitpy-0.7-py2.7-macosx-10.7-x86_64.egg/davitpy/models/tsyganenko/__init__.py:495: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  xx = self.xTrace[ip,0:self.l[ip]]
/Users/jc3e14/anaconda/lib/python2.7/site-packages/davitpy-0.7-py2.7-macosx-10.7-x86_64.egg/davitpy/models/tsyganenko/__init__.py:520: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  yy = self.zTrace[ip,0:self.l[ip]]
/Users/jc3e14/anaconda/lib/python2.7/site-packages/davitpy-0.7-py2.7-macosx-10.7-x86_64.egg/davitpy/models/tsyganenko/__init__.py:530: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  indMask = sign*self.yTrace[ip,0:self.l[ip]] < 0

Thought I'd bring these to your attention :)

[Shirling-VT]

Hi,

You should use "pip install --upgrade pillow" instead of "pip install --upgrade PIL" for pillow installation. Also, the brew installation script is not well tested. You'd better use the mac_port installation script for OS X dependency installation of davitpy. Or installing all dependencies listed in python_install_mac_port.sh which is updated instead of the dependencies listed in python_install_mac_brew.sh.

For your second problem, I don't see any tsygFortmodule.c file existing in my davitpy/models/tsyganenko/ directory. Either you didn't install gfortran properly (see python_install_mac_port.sh for installation) or you use old davitpy version? You can try to remove all the old davitpy files and "git clone https://github.com/vtsuperdarn/davitpy.git" a brand new davitpy and reinstall all the dependencies listed in python_install_mac_port.sh to see if it works.

Hope this helps.

Best wishes, Xueling

On Thu, Aug 3, 2017 at 9:26 AM, johncoxon notifications@github.com wrote:

I'm trying to get davitpy back on my Mac after quite a long time spent away. I manually installed every dependency listed in python_install_mac_brew.sh using brew and pip (skipping the line that installs Python, as I'm happy with my Anaconda install). PIL gave the following error:

Collecting pil Could not find a version that satisfies the requirement pil (from versions: ) No matching distribution found for pil

Everything else worked fine, but when I went to run python setup.py install I get the following error:

building extension "tsygFort" sources target build/src.macosx-10.7-x86_64-2.7/davitpy/models/tsyganenko/tsygFortmodule.c does not exist: Assuming tsygFortmodule.c was generated with "build_src --inplace" command. error: 'davitpy/models/tsyganenko/tsygFortmodule.c' missing

Any ideas why this is occurring?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/vtsuperdarn/davitpy/issues/316, or mute the thread https://github.com/notifications/unsubscribe-auth/AIEZK2EGvMfZB8fx_DVIej5xY8Cxwks2ks5sUcp9gaJpZM4OscAk .

-- Best wishes, Xueling

[johncoxon]

Hi Xueling, sorry, the two problems that you saw in the GitHub email are both problems I solved (I'd missed a line in the brew installs) and edited out of the issue – the new issue reflects a couple of problems that are actually present in davitpy. Thanks for replying though!

[asreimer]

Thanks @johncoxon, this looks like 2 problems: 1) Coefficients for tsyganenko model need updating 2) Code that doesn't make sure indicies are integers, which will break in newer versions of numpy

[johncoxon]

I've just tried re-running the above code, and what now happens is:

In [1]: import davitpy.models.tsyganenko as tsy
ERROR:root:Could not connect to remote DB: mongodb://sd_dbread:5d@sd-work9.ece.vt.edu:27017/radarInfo
WARNING:root:Reading hdw.dat info from local files in None
ERROR:root:radarRead: cannot read None
Traceback (most recent call last):
  File "/anaconda2/lib/python2.7/site-packages/davitpy-0.8-py2.7-macosx-10.6-x86_64.egg/davitpy/pydarn/radar/radInfoIo.py", line 67, in radarRead
    file_net = open(pathOpen, 'r')
TypeError: coercing to Unicode: need string or buffer, NoneType found
You may be getting this error because your computer cannot contact an appropriate internet server to get the latest radar.dat information.  You can use a local file instead by setting the SD_RADAR environment variable to the location of a local copy of radar.dat.

Example, you might add a similar line to your .bashrc:
export SD_RADAR=/home/username/tables/radar.dat

Also, make sure your SD_HDWPATH also points to the location of your hdw.dat files.
You can get the latest hdw.dat files from  https://github.com/vtsuperdarn/hdw.dat
Example, you might add a similar line to your .bashrc:
export SD_HDWPATH=/home/username/tables/hdw.dat/

ERROR:root:Could not update .radars.sqlite file with                               hdw.dat info

In [2]: lats = np.arange(10,90,10)

In [3]: lons = np.zeros_like(lats)

In [4]: rhos = 6372. * np.ones_like(lats)

In [5]: trace = tsy.tsygTrace(lats, lons, rhos)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-b4449b226f56> in <module>()
----> 1 trace = tsy.tsygTrace(lats, lons, rhos)

/anaconda2/lib/python2.7/site-packages/davitpy-0.8-py2.7-macosx-10.6-x86_64.egg/davitpy/models/tsyganenko/__init__.pyc in __init__(self, lat, lon, rho, filename, coords, datetime, vswgse, pdyn, dst, byimf, bzimf, lmax, rmax, rmin, dsmax, err)
    129         from datetime import datetime as pydt
    130 
--> 131         assert (None not in [lat, lon, rho]) or filename, 'You must provide either (lat, lon, rho) or a filename to read from'
    132 
    133         if None not in [lat, lon, rho]:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

In [6]: trace
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-6-c96da292b7dd> in <module>()
----> 1 trace

NameError: name 'trace' is not defined

In [7]: np.__version__
Out[7]: '1.13.3'

[ksterne]

Seems like there's an issue with in there with your .radars.sqlite file. Have you checked the couple of issues in the error message? Do you have local hdw.dat files? If not, it's possible your IP address has been blocked by sd-work9. Can you send it to me and I'll check the status?

[johncoxon]

@ksterne I have tried connecting to mongodb://sd_dbread:5d@sd-work9.ece.vt.edu:27017/radarInfo in the browser but it just tells me macOS can't parse the mongodb:// address. I don't have local files, because I don't really work with radar data (yet). My IP address is 2001:630:d0:8e0f:8c03:e1c1:9f1e:caba.

Any thoughts on the Tsyganenko module failing?

[ksterne]

That's strange on the parsing. Possibly the browser can't handle the mongodb protocol? I've never tried that. Let's stick with IPV4 address for now, can you pass that along (my e-mail or not github is also good here).

Not sure on Tsyganenko, but it could start with the hdw.dat database.

[johncoxon]

I believe my IPv4 is 152.78.193.202.

I get the message 'macOS doesn't recognise Internet addresses starting with mongodb://' when I put it into Safari. Typing it into Chrome brings up a Google search with this GitHub issue as the top result…

[ksterne]

OK, it doesn't look like your IP address has been picked up to be blocked here.

As with other issues relating to the hdw.dat mongodb (you might want to do a search of the current open and closed issues), can you try from a command line prompt:

nc -vz sd-work9.ece.vt.edu 27017

and send back the output. If this does not succeed, then there is an issue between your computer and our server (likely start is checking with your ISP to see if that port is blocked). If you get a message that says succeeded, then you might try what's been suggested in #339.

[johncoxon]

john$ nc -vz sd-work9.ece.vt.edu 271017
nc: port range not valid

I must confess a six-digit port seems odd to me. Shouldn't it be five-digit?

[ksterne]

Whoops, case of early morning, (tired) fat fingers. Port number should match the one you're seeing in the error messages. I've edited my message to correct the port number.

[johncoxon]

That generated:

john$ nc -vz sd-work9.ece.vt.edu 27017
found 0 associations
found 1 connections:
     1: flags=82<CONNECTED,PREFERRED>
    outif en7
    src 152.78.193.202 port 65049
    dst 128.173.144.41 port 27017
    rank info not available
    TCP aux info available

Connection to sd-work9.ece.vt.edu port 27017 [tcp/*] succeeded!

But I don't understand, then, why the line ERROR:root:Could not connect to remote DB: mongodb://sd_dbread:5d@sd-work9.ece.vt.edu:27017/radarInfo appears at the top of the error. Clearly something is going wrong there.

[johncoxon]

So I was looking into this and I think it's because os.getenv('SD_RADAR') returns None on line 63 of radInfolo.py. This is true on my laptop running davitpy 0.7 and running davitpy 0.8. The reason I only noticed it on my new laptop is because radInfolo.radarRead is not run when I do from davitpy.models import aacgm, tsyganenko in v0.7 but it is run when I do the same in 0.8, which is why this failure message only crops up now. (Not sure why the mongodb is not connecting, but at least it not connecting doesn't seem to be something new in v0.8.)

On the one hand I can appreciate that, since davitpy is primarily a python package aimed at people using SuperDARN, it makes sense to import the contents of pydarn by default. On the other hand, I'm not using that part of davitpy and I've not imported it, so I would probably lean towards thinking it's a good idea to revert to the v0.7 behaviour, as this message just clutters up my IPython notebooks with no actual benefit to me.

[johncoxon]

I have now made progress on both of these bugs. The bug in tsyganenko was fixed by @asreimer in #358 (accepting this pull request would mean we could close this issue! Please consider it…)

I plan to open a new issue for the mongodb bug since this issue has a very generic name which isn't actually all that helpful.

[ksterne]

Now that #358 is merged into develop, closing this issue. Sorry it was such a long time coming.