This pull request fixes Synda's behavior in two situations where it has crashed recently.
In issue 152 is a report of an event in which garbled data from an index node caused an exception in sdfiledao.add_file(). It was not caught, so Synda crashed. The changes in sdenqueue.py catch any exception form sdfiledao.add_file(), log it, and continue to the next file if any. To make the log useful, we need to print the search url which led to the problem. In order to make it available, I added this to the file transfer object; that is a change in sdnetutils.py.
After a recent change to ESGF software, we no longer have a guarantee that a search query will return facet_counts. Now it is returned only "if the user requests specific, comma separated field names, using the “facets” URL parameter. " This change is presently local to LLNL, but we can expect it to become universal in the near future (it prevents index node failures when traffic is high). So Synda has to function without crashing when facet_counts is not returned. Fortunately, the changes required are minimal.
This pull request fixes Synda's behavior in two situations where it has crashed recently.
In issue 152 is a report of an event in which garbled data from an index node caused an exception in sdfiledao.add_file(). It was not caught, so Synda crashed. The changes in sdenqueue.py catch any exception form sdfiledao.add_file(), log it, and continue to the next file if any. To make the log useful, we need to print the search url which led to the problem. In order to make it available, I added this to the file transfer object; that is a change in sdnetutils.py.
After a recent change to ESGF software, we no longer have a guarantee that a search query will return facet_counts. Now it is returned only "if the user requests specific, comma separated field names, using the “facets” URL parameter. " This change is presently local to LLNL, but we can expect it to become universal in the near future (it prevents index node failures when traffic is high). So Synda has to function without crashing when facet_counts is not returned. Fortunately, the changes required are minimal.