Open danstoner opened 7 years ago
I believe the behavior has changed from original report, possibly due to commit: b48cff25527b0ef6321e21d4631eea796eb2db22
I believe without the parserver (I cannot find where or how it is supposed to be running, but I think it is ruby code somewhere), on a machine where nothing is listening at localhost:4334, the code does not display an error but instead returns only empty results.
~/idb-backend/idb/helpers# ipython
Python 2.7.12 (default, Nov 20 2017, 18:23:56)
Type "copyright", "credits" or "license" for more information.
IPython 5.1.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: from biodiversity_socket_connector import Biodiversity
In [2]: b = Biodiversity()
In [3]: b.get_genus_species("Puma concolor")
No handlers could be found for logger "idb.cfg"
Out[3]: {}
The logging handler issue appears to be masking the underlying error, which is failure to connect to socket server.
In [7]: b.get_genus_species("Puma concolor")
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-7-c8b39bae692c> in <module>()
----> 1 b.get_genus_species("Puma concolor")
/home/dstoner/git/idb-backend/idb/helpers/biodiversity_socket_connector.py in get_genus_species(self, namestr)
90
91 def get_genus_species(self, namestr):
---> 92 self._sendOne(namestr)
93 return self._parseResp(self._recvOne())
94
/home/dstoner/git/idb-backend/idb/helpers/biodiversity_socket_connector.py in _sendOne(self, namestr)
40
41 def _sendOne(self, namestr):
---> 42 if self.sock:
43 try:
44 self.sock.send(namestr.encode("utf-8") + "\n")
/home/dstoner/git/idb-backend/idb/helpers/biodiversity_socket_connector.py in sock(self)
27 def sock(self):
28 if self._sock is None:
---> 29 self._connect()
30 return self._sock
31
/home/dstoner/git/idb-backend/idb/helpers/biodiversity_socket_connector.py in _connect(self)
21 self._sock.connect((self.host, self.port))
22 except socket.error as e:
---> 23 logger.error("Biodiversity socket server unavailable: %s", e)
24 self._sock = False
25
NameError: global name 'logger' is not defined
The missing socket connection no longer causes an indefinite block on index processes, but is the current behavior acceptable? Is the data from the socket actually needed?
On the indexing / workflow server, the parserver service is not running and is not enabled... so indexing has been hitting this error for one while.
Cannot tell if this has affected indexing or not. The indexer process seems to run to completion just fine but is it doing all of the work it is supposed to be doing?
# systemctl list-units | grep parserver
# systemctl status parserver
● parserver.service - Biodiversity parserver
Loaded: loaded (/etc/systemd/system/parserver.service; disabled; vendor preset: enabled)
Active: inactive (dead)
# systemctl start parserver
# systemctl status parserver
● parserver.service - Biodiversity parserver
Loaded: loaded (/etc/systemd/system/parserver.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2018-04-20 10:51:31 EDT; 1s ago
Main PID: 15853 (parserver)
CGroup: /system.slice/parserver.service
└─15853 /usr/bin/ruby2.3 /usr/local/bin/parserver
Apr 20 10:51:31 c18node4 systemd[1]: Started Biodiversity parserver.
# systemctl list-unit-files | grep parserver
parserver.service disabled
# systemctl stop parserver
# cat /etc/systemd/system/parserver.service
[Unit]
Description=Biodiversity parserver
[Service]
User=biodiversity
WorkingDirectory=~
ExecStart=/usr/local/bin/parserver
Restart=always
RestartSec=2min
[Install]
WantedBy=multi-user.target
Biodiversity() is used in conversions.py.
b = Biodiversity()
def genusSpeciesFiller(t, r):
gs = b.get_genus_species(r["scientificname"])
return gs
def gs_sn_crossfill(t, r):
if filled("genus", r) and not filled("scientificname", r):
r["scientificname"] = scientificNameFiller(t, r)
r["flag_scientificname_added"] = True
elif filled("scientificname", r) and not filled("genus", r):
gs = genusSpeciesFiller(t, r)
for k, indk in [("genus", "genus"), ("species", "specificepithet")]:
if filled(k, gs) and not filled(indk, r):
r[indk] = gs[k]
r["flag_" + indk + "_added"] = True
There is currently no logging or debug logging in conversions for these functions. gs is most likely coming back as empty dict when Biodiversity / parserver is not actually running.
Changed "b" variable to "bioserv" in c2d5991d5afdff9987d537acc41654b78b6a355c
$ find . -name "*.py" | xargs grep -C 1 bioserv
./idb/helpers/conversions.py-
./idb/helpers/conversions.py:bioserv = Biodiversity()
./idb/helpers/conversions.py-
--
./idb/helpers/conversions.py-def genusSpeciesFiller(t, r):
./idb/helpers/conversions.py: gs = bioserv.get_genus_species(r["scientificname"])
./idb/helpers/conversions.py- return gs
The idb indexer does not seem to have a timeout while connecting to the ruby Biodiversity parserver.
For example, when running:
idb index check
If the parserver is not responding, the indexer process gets stuck waiting "forever" for a response.