search

EOL / tramea

A lightweight server for denormalized EOL data
Other
2 stars 1 forks source link

strange symptoms in resource 223 #304

Open jhammock opened 8 years ago

jhammock commented 8 years ago

In this resource http://eol.org/resources/223

two odd things are happening:

the first 10% of the images tab of the resource collection (sorted by recently added) is paginated into short pages, beginning with only a few images per page, and growing longer until the normal number appears about 10% of the way through.

This by itself is a very minor issue, mentioning it here in case it is related to the other issue.

A large number of the maps (possibly half) show no points, where point are visible on the source maps. Several examples here

This is a connector, which has Been Through Some Stuff, having supplied the detail tab and then been moved to the maps tab. @eliagbayani , if you can access it, could you run the connector and see what the resource looks like now? Maybe a reharvest will fix this...

eliagbayani commented 8 years ago

I was able to run the connector from the server and generate the latest XML resource. The numbers in the current collection seems incorrect. http://eol.org/collections/320

Below are the numbers I got from the latest generated XML resource and one from July 2014.

Taxa ---------- Maps 541,787 ---------- 589,225 24-Jul-14 597,942 ---------- 597,942 29-Jun-16

Resource is now set to force-harvest. We should just re-harvest it and see how it goes.

eliagbayani commented 8 years ago

I’ve set this up to force-harvest a couple of days ago. Now, resource says ‘Harvest Failed’. This is copy of the resource file. I copied this from our server, generated by the connector. Resource validates OK using our validator. I used my local validator since file is too big to run remotely.

I’m inclined to say that maybe harvesting may still not be ready to handle such big resource.

JRice commented 8 years ago

Harvesting at this point is NOT automated, and any "Force Harvest" that is set will end up "Harvest Failed" the following day unless I add it to a whitelist in the PHP code.

We're really being very careful about what goes through since the port of the publishing code.

It's nothing personal! ;)

Jen can take your ID and add it to the (manually-maintained) queue...

On Thu, Jul 7, 2016 at 1:29 AM, eliagbayani notifications@github.com wrote:

I’ve set this up to force-harvest a couple of days ago. Now, resource http://eol.org/content_partners/311/resources/223 says ‘Harvest Failed’. This is copy of the resource file https://dl.dropboxusercontent.com/u/5763406/resources/223.xml.gz. I copied this from our server, generated by the connector. Resource validates OK using our validator http://services.eol.org/validator/. I used my local validator since file is too big to run remotely.

I’m inclined to say that maybe harvesting may still not be ready to handle such big resource.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/EOL/tramea/issues/304#issuecomment-230982579, or mute the thread https://github.com/notifications/unsubscribe/AABebtz49hlbEfOuikV7xdoD58rlwCYrks5qTI6lgaJpZM4IoinD .

jhammock commented 8 years ago

Ohhh... Upload succeeded, then? Now I think I understand the upload segment of things. This resource is in the queue, not very high since I'm not sure when we'll feel ready for 600k taxa :) Here's hoping the symptoms disappear upon reharvest..