number of records - Githubissues

plazi / treatmentBank

Repository devoted to house keeping of treatmentBank

0 stars 0 forks source link

number of records #6

Open punkish opened 3 years ago

punkish commented 3 years ago

hi @gsautter

last night I downloaded all the archives (full, monthly, weekly, and daily) and expanded them into a directory. I got the following numbers

archive	cumulative num of treatments
plazi.zenodeo.zip	613867
plazi.zenodeo.monthly.zip	616191
plazi.zenodeo.weekly.zip	618784
plazi.zenodeo.daily.zip	623313

I inserted the data into my db and got the following numbers

{
    "treatments": 623313,
    "treatmentCitations": 562575,
    "treatmentAuthors": 2251170,
    "materialsCitations": 1058635,
    "collectionCodes": 348697,
    "figureCitations": 1698987,
    "bibRefCitations":2734988
}

But when I look at the stats on plazi.org, I see Treatments: 715061. What is the explanation for that almost 92K discrepancy in the number of treatments? What should I get and what should be reported on plazi.org?

cc @myrmoteras

gsautter commented 3 years ago

That discrepancy is simply because the dumps do not include ZooBank stubs, i.e., placeholders that indicate that according to ZooBank, there must be a treatment, but we don't have it (yet), and all we know is the taxon name and the metadata.

deepreef commented 3 years ago

@gsautter : I'm currently working to improve ZooBank services, and have been thinking a lot about how useful it would be to get back to coordinating efforts again (the lapse is 100% my fault -- just too distracted by other things!)

I can think of several things that could help:

Editorial access to ZooBank so that non-public records (and their identifiers) can be discovered.
Better apis to allow you to discover when a treatment and its associated publication have been registered.
Ability to update ZooBank records that are still recorded as "In press", after they are published and discovered by PLAZI.
Ability to insert non-nomenclatural act treatments (i.e., subsequent usages) into GNUB (database behind ZooBank)

I can't promise hundreds of hours of time, but if there are things that would help the integration process I can certainly start developing those things.

punkish commented 3 years ago

That discrepancy is simply because the dumps do not include ZooBank stubs, i.e., placeholders that indicate that according to ZooBank, there must be a treatment, but we don't have it (yet), and all we know is the taxon name and the metadata.

ok, thanks for the explanation.

My suggestion would be to modify the stats js (pubAndTreatmentCounts, pubAndTreatmentCountsWithStubs, materialsCitationCount, materialsCitationGeoRefCount) so the numbers are the same all across the board. That is, to not include the placeholders until you actually get the treatment. Having different numbers is confusing and a user can start wondering why TB is showing a different number from Zenodeo or other websites/apps that might use the XML data.

cc @myrmoteras