search

NYULibraries / hsl2

Repository for the HSL phase 2 project
1 stars 1 forks source link

Updates to ERD Extract / Cron Job to BobCat Dev #6

Closed mahers01 closed 10 years ago

mahers01 commented 10 years ago

Last week I added a new record to ERD to see if:

1) the record was captured in the ERD extract, and 2) the cron job of the extract would update our holdings in BobCat Dev

When I checked this morning the new record (Record ID 859302; "GIT HUB CRON JOB") did not appear in BobCat Dev (http://bobcatdev.library.nyu.edu/primo_library/libweb/action/search.do?vid=HSL)

chrpr commented 10 years ago

Thanks Stephen,

Looks like it's actually been failing a bit longer than that. Did something change at any point with regard to character encoding?

My script is throwing a "UnicodeDecodeError" (details below). It's doing the same thing on my local machine, which it never had during any of our initial testing. I can take a look at debugging again, but not until tomorrow or Friday at the earliest. I am concerned that the data extracts were unicode in the past, but now appear not to be.

Thanks, -Corey UnicodeDecodeError: 'utf8' codec can't decode byte 0xbc in position 0: invalid start byte Build step 'Execute shell' marked build as failure Finished: FAILURE

ianlamb77 commented 10 years ago

Hey guys, thanks for noticing this! I'm not sure how the encoding was handled in the past but I am looking into it now.

Regards, Ian

ianlamb77 commented 10 years ago

Hi all, I've enforced UTF-8 encoding in the call to the mysql client, which resulted in 5,288 insertions/deletions the next time I pushed it to github...so it definitely did something. Let me know if that helped on your end.

Thanks, Ian

scotdalton commented 10 years ago

@ianlamb77, it did something on our end as well. We think your change fixed it (it definitely fixed the jenkins job), but need to wait for the Primo pipe to run this evening to be certain. We'll keep you posted.

Thanks, Scot

chrpr commented 10 years ago

Looks like this resolved the issue. Thanks for addressing it so quickly, Ian. It's worth noting that there are a few thousand fewer records in this batch than there have been in the past few, but if someone wants to keep track of & look into that I recommend doing so on another issue.

"GIT HUB CRON JOB" is now appearing in your search results.

Marking as closed.

-Corey