Closed nataliep closed 4 years ago
Hi Natalie,
It's good to see you. I've set up my environment to match yours: (with python3.7.1 and the newest bs4, requests, etc).
Is the "UNM LINK $0" searchfield is a custom made field?
Does the page work in the browser? I.e., if you go to Enterprise Admin page > Search Configuration > Search Fields > "UNM LINK $0" > Edit ------------ does it work like expected?
Garrett
Garrett,
It’s a delivered MARC map. We have 25 MARC maps for UNIMARC (Universal MARC format) tags.
UNM LINK $0 UNM LINK $7 UNM LINK $A UNM LINK $B UNM LINK $C UNM LINK $D UNM LINK $E UNM LINK $F UNM LINK $G UNM LINK $H UNM LINK $I UNM LINK $L UNM LINK $N UNM LINK $O UNM LINK $P UNM LINK $S UNM LINK $T UNM LINK $U UNM LINK $X UNM LINK $Y UNM LINK $Z UNM LINK DISPLAY 412 UNM LINK DISPLAY 488 UNM LNK 412 $0 UNM LNK 488 $0
I have no use for those MARC maps. I don’t know which SirsiDynix customers would need them. The link to configure the first UNIMARC tag listed above is https://lsulaw.ent.sirsi.net/client/en_US/admin/search/managemarcmaps.confunimarctags/UNM$0020LINK$0020$$0/true.
Natalie
From: garrett armstrong notifications@github.com Sent: Thursday, February 28, 2019 11:25 AM To: lsulibraries/SirsiDynexFunctionalTests SirsiDynexFunctionalTests@noreply.github.com Cc: Natalie Palermo palermo@lsu.edu; Author author@noreply.github.com Subject: Re: [lsulibraries/SirsiDynexFunctionalTests] Exceptions (#2)
Hi Natalie,
It's good to see you. I've set up my environment to match yours: (with python3.7.1 and the newest bs4, requests, etc).
Is the "UNM LINK $0" searchfield is a custom made field? Does the page work in the browser? I.e., if you go to Enterprise Admin page > Search Configuration > Search Fields > "UNM LINK $0" > Edit ------------ does it work like expected?
Garrett
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/lsulibraries/SirsiDynexFunctionalTests/issues/2#issuecomment-468360988, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AtzuAYsRkFRcXAy5oXToLZ-zNStmHJvFks5vSBD9gaJpZM4bTKYV.
Aha! You found the problem.
Since we don't use UNIMARC at Middleton, the script doesn't know what to do with them. Maybe the script should skip all marcmaps that begin with "UNM LINK ".
I'm ok with that. Do you plan to make two scripts available in the Developers Community or just note that the one script excludes UNIMARC?
@nataliep On second thought, I believe the first solution was a red herring.
What stands out to me is a missing $ in the url. Your working link is: https://lsulaw.ent.sirsi.net/client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$$0
while the script is genereting a bad url with a missing $ toward the end: https://lsulaw.ent.sirsi.net/client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$0
The script should be doing this correctly. Let's try updating your version of the script to the latest version. Either git pull origin
from within your SirsiDynexFunctinoalTests folder, or manually copy/paste this site's scrape_Enterprise_searchconfig_marcmaps.py over your computer's version.
If that doesn't work, let me know so we can keep ploblem solving, Garrett
@nataliep
With your Enterprise admin open in a browser & with yourself logged in, enter each of these urls: https://lsulaw.ent.sirsi.net/client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$0 https://lsulaw.ent.sirsi.net/client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$$0
Let me know which one works, garrett
(Side note: I renamed the script & moved it to a better location. https://github.com/lsulibraries/ScrapingWebpages/blob/master/SirsiEnterprise/export_marcmaps_searchfields/export_marcmaps_searchfields.py)
The second url works.
Hi @nataliep,
Let's assume your version of the code is out of date. Try this --> Delete your scrape_Enterprise_searchconfig_marcmaps.py file and get the newest version from: https://github.com/lsulibraries/ScrapingWebpages/blob/master/SirsiEnterprise/export_marcmaps_searchfields/export_marcmaps_searchfields.py
when you run python export_marcmaps_searchfields.py
, I expect the script to complete without errors.
Let me know what happens, Garrett
No luck.
https://lsulaw.ent.sirsi.net/client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$0 Traceback (most recent call last): File "C:\ProgramData\Anaconda3\lib\site-packages\requests\adapters.py", line 449, in send timeout=timeout File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 731, in urlopen body_pos=body_pos, response_kw) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 731, in urlopen body_pos=body_pos, response_kw) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 731, in urlopen body_pos=body_pos, **response_kw) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 711, in urlopen retries = retries.increment(method, url, response=response, _pool=self) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\util\retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='lsulaw.ent.sirsi.net', port=443): Max retries exceeded with url: /client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$0 (Caused by ResponseError('too many 500 error responses'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "export_marcmaps_searchfields.py", line 301, in
I've extended the number of retries. Can you pull the newest version from https://github.com/lsulibraries/ScrapingWebpages/blob/master/SirsiEnterprise/export_marcmaps_searchfields/export_marcmaps_searchfields.py
If it errors out again, can you run it with python export_marcmaps_searchfields --verbose
and send the last 100 or so line of the error message.
Thanks
See attached file. errors.txt
Enterprise Version 4.5 (36068) Python 3.7.1
Search fields were scraped just fine. Keeps crashing at the following URL:
https://lsulaw.ent.sirsi.net/client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$0 Traceback (most recent call last): File "C:\ProgramData\Anaconda3\lib\site-packages\requests\adapters.py", line 449, in send timeout=timeout File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 731, in urlopen body_pos=body_pos, response_kw) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 731, in urlopen body_pos=body_pos, response_kw) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 731, in urlopen body_pos=body_pos, **response_kw) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 711, in urlopen retries = retries.increment(method, url, response=response, _pool=self) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\util\retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='lsulaw.ent.sirsi.net', port=443): Max retries exceeded with url: /client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$0 (Caused by ResponseError('too many 500 error responses'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "scrape_Enterprise_searchconfig_marcmaps.py", line 301, in
do_marcmaps()
File "scrape_Enterprise_searchconfig_marcmaps.py", line 165, in do_marcmaps
i: get_marc_info(i) for i in all_marc_codes
File "scrape_Enterprise_searchconfig_marcmaps.py", line 165, in
i: get_marc_info(i) for i in all_marc_codes
File "scrape_Enterprise_searchconfig_marcmaps.py", line 184, in get_marc_info
marcmap_item_dict = parse_marcmap_item_details(marc_code)
File "scrape_Enterprise_searchconfig_marcmaps.py", line 197, in parse_marcmap_item_details
marcmap_item_details_soup = make_soup('managemarcmaps.edit', marc_code)
File "scrape_Enterprise_searchconfig_marcmaps.py", line 38, in make_soup
page = requests_retry_session(session=s).get(full_url)
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py", line 546, in get
return self.request('GET', url, kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, send_kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\adapters.py", line 507, in send
raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='lsulaw.ent.sirsi.net', port=443): Max retries exceeded with url: /client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$0 (Caused by ResponseError('too many 500 error responses'))