Closed nataliep closed 4 years ago
GarrettArm
commented
5 years ago Hi Natalie,
It's good to see you. I've set up my environment to match yours: (with python3.7.1 and the newest bs4, requests, etc).
Is the "UNM LINK $0" searchfield is a custom made field?
Does the page work in the browser? I.e., if you go to Enterprise Admin page > Search Configuration > Search Fields > "UNM LINK $0" > Edit ------------ does it work like expected?
Garrett
nataliep
commented
5 years ago Garrett,
It’s a delivered MARC map. We have 25 MARC maps for UNIMARC (Universal MARC format) tags.
UNM LINK $0 UNM LINK $7 UNM LINK $A UNM LINK $B UNM LINK $C UNM LINK $D UNM LINK $E UNM LINK $F UNM LINK $G UNM LINK $H UNM LINK $I UNM LINK $L UNM LINK $N UNM LINK $O UNM LINK $P UNM LINK $S UNM LINK $T UNM LINK $U UNM LINK $X UNM LINK $Y UNM LINK $Z UNM LINK DISPLAY 412 UNM LINK DISPLAY 488 UNM LNK 412 $0 UNM LNK 488 $0
I have no use for those MARC maps. I don’t know which SirsiDynix customers would need them. The link to configure the first UNIMARC tag listed above is https://lsulaw.ent.sirsi.net/client/en_US/admin/search/managemarcmaps.confunimarctags/UNM$0020LINK$0020$$0/true.
Natalie
From: garrett armstrong notifications@github.com Sent: Thursday, February 28, 2019 11:25 AM To: lsulibraries/SirsiDynexFunctionalTests SirsiDynexFunctionalTests@noreply.github.com Cc: Natalie Palermo palermo@lsu.edu; Author author@noreply.github.com Subject: Re: [lsulibraries/SirsiDynexFunctionalTests] Exceptions (#2)
Hi Natalie,
It's good to see you. I've set up my environment to match yours: (with python3.7.1 and the newest bs4, requests, etc).
Is the "UNM LINK $0" searchfield is a custom made field? Does the page work in the browser? I.e., if you go to Enterprise Admin page > Search Configuration > Search Fields > "UNM LINK $0" > Edit ------------ does it work like expected?
Garrett
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/lsulibraries/SirsiDynexFunctionalTests/issues/2#issuecomment-468360988, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AtzuAYsRkFRcXAy5oXToLZ-zNStmHJvFks5vSBD9gaJpZM4bTKYV.
GarrettArm
commented
5 years ago Aha! You found the problem.
Since we don't use UNIMARC at Middleton, the script doesn't know what to do with them. Maybe the script should skip all marcmaps that begin with "UNM LINK ".
nataliep
commented
5 years ago I'm ok with that. Do you plan to make two scripts available in the Developers Community or just note that the one script excludes UNIMARC?
GarrettArm
commented
5 years ago @nataliep On second thought, I believe the first solution was a red herring.
What stands out to me is a missing $ in the url. Your working link is: https://lsulaw.ent.sirsi.net/client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$$0
while the script is genereting a bad url with a missing $ toward the end: https://lsulaw.ent.sirsi.net/client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$0
The script should be doing this correctly. Let's try updating your version of the script to the latest version. Either git pull origin from within your SirsiDynexFunctinoalTests folder, or manually copy/paste this site's scrape_Enterprise_searchconfig_marcmaps.py over your computer's version.
If that doesn't work, let me know so we can keep ploblem solving, Garrett
GarrettArm
commented
5 years ago @nataliep
With your Enterprise admin open in a browser & with yourself logged in, enter each of these urls: https://lsulaw.ent.sirsi.net/client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$0 https://lsulaw.ent.sirsi.net/client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$$0
Let me know which one works, garrett
(Side note: I renamed the script & moved it to a better location. https://github.com/lsulibraries/ScrapingWebpages/blob/master/SirsiEnterprise/export_marcmaps_searchfields/export_marcmaps_searchfields.py)
nataliep
commented
5 years ago The second url works.
GarrettArm
commented
5 years ago Hi @nataliep,
Let's assume your version of the code is out of date. Try this --> Delete your scrape_Enterprise_searchconfig_marcmaps.py file and get the newest version from: https://github.com/lsulibraries/ScrapingWebpages/blob/master/SirsiEnterprise/export_marcmaps_searchfields/export_marcmaps_searchfields.py
when you run python export_marcmaps_searchfields.py, I expect the script to complete without errors.
Let me know what happens, Garrett
nataliep
commented
5 years ago No luck.
https://lsulaw.ent.sirsi.net/client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$0 Traceback (most recent call last): File "C:\ProgramData\Anaconda3\lib\site-packages\requests\adapters.py", line 449, in send timeout=timeout File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 731, in urlopen body_pos=body_pos, response_kw) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 731, in urlopen body_pos=body_pos, response_kw) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 731, in urlopen body_pos=body_pos, **response_kw) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 711, in urlopen retries = retries.increment(method, url, response=response, _pool=self) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\util\retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='lsulaw.ent.sirsi.net', port=443): Max retries exceeded with url: /client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$0 (Caused by ResponseError('too many 500 error responses'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "export_marcmaps_searchfields.py", line 301, in
GarrettArm
commented
5 years ago I've extended the number of retries. Can you pull the newest version from https://github.com/lsulibraries/ScrapingWebpages/blob/master/SirsiEnterprise/export_marcmaps_searchfields/export_marcmaps_searchfields.py
If it errors out again, can you run it with python export_marcmaps_searchfields --verbose and send the last 100 or so line of the error message.
Thanks
nataliep
commented
5 years ago See attached file. errors.txt
Enterprise Version 4.5 (36068) Python 3.7.1
Search fields were scraped just fine. Keeps crashing at the following URL:
https://lsulaw.ent.sirsi.net/client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$0 Traceback (most recent call last): File "C:\ProgramData\Anaconda3\lib\site-packages\requests\adapters.py", line 449, in send timeout=timeout File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 731, in urlopen body_pos=body_pos, response_kw) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 731, in urlopen body_pos=body_pos, response_kw) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 731, in urlopen body_pos=body_pos, **response_kw) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 711, in urlopen retries = retries.increment(method, url, response=response, _pool=self) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\util\retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='lsulaw.ent.sirsi.net', port=443): Max retries exceeded with url: /client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$0 (Caused by ResponseError('too many 500 error responses'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "scrape_Enterprise_searchconfig_marcmaps.py", line 301, in
do_marcmaps()
File "scrape_Enterprise_searchconfig_marcmaps.py", line 165, in do_marcmaps
i: get_marc_info(i) for i in all_marc_codes
File "scrape_Enterprise_searchconfig_marcmaps.py", line 165, in
i: get_marc_info(i) for i in all_marc_codes
File "scrape_Enterprise_searchconfig_marcmaps.py", line 184, in get_marc_info
marcmap_item_dict = parse_marcmap_item_details(marc_code)
File "scrape_Enterprise_searchconfig_marcmaps.py", line 197, in parse_marcmap_item_details
marcmap_item_details_soup = make_soup('managemarcmaps.edit', marc_code)
File "scrape_Enterprise_searchconfig_marcmaps.py", line 38, in make_soup
page = requests_retry_session(session=s).get(full_url)
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py", line 546, in get
return self.request('GET', url, kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, send_kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\adapters.py", line 507, in send
raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='lsulaw.ent.sirsi.net', port=443): Max retries exceeded with url: /client/en_US/admin/search/managemarcmaps.edit/UNM$0020LINK$0020$0 (Caused by ResponseError('too many 500 error responses'))