search

yjunechoe / Snowglobe

Snowglobe
http://snowglobe.soc.northwestern.edu/
2 stars 0 forks source link

SQL Statements not working for fast.scrape #10

Closed mcweenysean closed 3 years ago

mcweenysean commented 3 years ago

I got the stage/fetch portion of the search working, but the search part itself is being poorly behaved. The fast.scrape() function that uses our SQL query is not working properly though. The dbGetQuery() syntax should be correct, I think the index created properly, as the results are returned quite quickly, but they were Null. If I do the search on Year, it takes forever, like expected without an index. Any ideas on what could be going on here?

mcweenysean commented 3 years ago

found the issue. all of the PaperIDs are 0. Not sure how that happened, but it means I will have to go back to campus to reupload everything.

yjunechoe commented 3 years ago

Huh that's really weird because afaik the indexed ID columns in REFS are fine, right? Maybe it's something wrong with the encoding? It might help to check the type of the columns in the REFS table as you uploaded it (character? double? integer?) and just convert the PaperID column of the PAPER_INFO table to the same type before uploading - just a guess.

mcweenysean commented 3 years ago

yeah the indexed refs works great - forward.search() and backward.search" and all of their derivative functions work great.

Both paperid columns are listed as double

mcweenysean commented 3 years ago

Turns out we can just restore to a previous db snapshot so we'll see how that goes.

mcweenysean commented 3 years ago

it either imported wrong or got fucked up before the last 7 days, so I will need to return to campus for an upload. Unfortunately, my mac is still broken so I'll have to do some planning.