DOAJ / doaj

The Directory of Open Access Journals - website and directory software
Apache License 2.0
59 stars 16 forks source link

2619 articles return 500 Internal Server error #964

Closed dommitchell closed 7 years ago

dommitchell commented 8 years ago

http://bit.ly/1OT479U

These are all article of the license type 'CC by-nc' which is an old license type and shouldn't be on the site. Note that there are 2 CC BY-NC listing under license now:

CC BY-NC (350137) CC by-nc (2619)

Any idea what is going on?

emanuil-tolev commented 8 years ago

A bit of an idea. It'll crystallise soon enough. cc @richard-jones @Steven-Eardley did we not have to deal with a similar error on article pages before?

If not we'll investigate more thoroughly (though I think we have enough info to replicate). Below is the error log, and below that, the article's JSON record in Elasticsearch so we can replicate locally.

Looks like we might need to escalate that job which will refresh article metadata based on the journal info - this article has issns so it's reachable via a journal ToC and the discovery API, but has no "journal" section in the JSON. This is I believe planned as part of the Harvester project since it deals with refreshing article metadata automatically too. Nonetheless, to fix this we could create a separate script from the current portality/scripts/journalinfo.py which looks for articles without a "journal" section or which have a "journal" but no "journal.bibjson" (just in case) - and updates them. Let me know if you have any thoughts about doing it differently or if I'm wrong about the cause of this.

2015-12-09 13:52:28,556 - portality.core - ERROR - Exception on /article/000c38480c3d4d1383444cf8b24bc85d [GET]
Traceback (most recent call last):
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/flask/app.py", line 1687, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/flask/app.py", line 1360, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/flask/app.py", line 1358, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/flask/app.py", line 1344, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/cloo/repl/production/doaj/src/doaj/portality/view/doaj.py", line 278, in article_page
    return render_template('doaj/article.html', article=article, journal=journal)
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/flask/templating.py", line 125, in render_template
    context, ctx.app)
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/flask/templating.py", line 107, in _render
    rv = template.render(context)
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/cloo/repl/production/doaj/src/doaj/portality/templates/doaj/article.html", line 1, in top-level template code
    {% extends "base.html" %}
  File "/home/cloo/repl/production/doaj/src/doaj/portality/templates/base.html", line 21, in top-level template code
    {% block extra_meta_tags %}{% endblock %}
  File "/home/cloo/repl/production/doaj/src/doaj/portality/templates/doaj/article.html", line 18, in block "extra_meta_tags"
    {% set jbib = journal.bibjson() %}
UndefinedError: 'None' has no attribute 'bibjson'
{
    "_index":"doaj",
    "_type":"article",
    "_id":"000c38480c3d4d1383444cf8b24bc85d",
    "_version":2,
    "found":true,
    "_source":{
        "index":{
            "publisher":[
                "Centro de Estudos de Venenos e Animais Pe\u00e7onhentos - CEVAP, Universidade Estadual Paulista - UNESP"
            ],
            "schema_subject":[
                "LCC:Medicine",
                "LCC:Zoology",
                "DOAJ:Health Sciences",
                "LCC:Internal medicine",
                "DOAJ:Biology and Life Sciences",
                "LCC:Arctic medicine. Tropical medicine",
                "DOAJ:Medicine (General)",
                "DOAJ:Internal medicine",
                "DOAJ:Biology",
                "DOAJ:Zoology",
                "LCC:Special situations and conditions",
                "LCC:Science"
            ],
            "license":[
                "CC by-nc"
            ],
            "classification":[
                "Internal medicine",
                "Biology and Life Sciences",
                "Biology",
                "Health Sciences",
                "Special situations and conditions",
                "Science",
                "Medicine (General)",
                "Arctic medicine. Tropical medicine",
                "Zoology",
                "Medicine"
            ],
            "classification_paths":[
                "Medicine: Internal medicine: Special situations and conditions: Arctic medicine. Tropical medicine",
                "Science: Zoology"
            ],
            "country":"Brazil",
            "issn":[
                "1678-4936",
                "0104-7930"
            ],
            "unpunctitle":"PRELIMINARY INVESTIGATION ABOUT THE ANTICOMPLEMENT PROPERTIES OF PARTIALLY SEPARATED Vipera ammodytes VENOM",
            "language":[
                "English"
            ],
            "date":"1997-01-01T00:00:00Z",
            "schema_code":[
                "LCC:RC955-962",
                "LCC:RC952-1245",
                "LCC:RC31-1245",
                "LCC:R",
                "LCC:Q",
                "LCC:QL1-991"
            ],
            "asciiunpunctitle":"PRELIMINARY INVESTIGATION ABOUT THE ANTICOMPLEMENT PROPERTIES OF PARTIALLY SEPARATED Vipera ammodytes VENOM",
            "subject":[
                "Internal medicine",
                "Biology and Life Sciences",
                "Biology",
                "Health Sciences",
                "Special situations and conditions",
                "Science",
                "Medicine (General)",
                "Arctic medicine. Tropical medicine",
                "Zoology",
                "Medicine"
            ]
        },
        "last_updated":"2015-04-22T01:21:44Z",
        "created_date":"2013-08-31T05:26:16Z",
        "admin":{
            "in_doaj":true
        },
        "id":"000c38480c3d4d1383444cf8b24bc85d",
        "bibjson":{
            "start_page":"37",
            "title":"PRELIMINARY INVESTIGATION ABOUT THE ANTI-COMPLEMENT PROPERTIES OF PARTIALLY SEPARATED Vipera ammodytes VENOM",
            "journal":{
                "publisher":"Centro de Estudos de Venenos e Animais Pe\u00e7onhentos - CEVAP, Universidade Estadual Paulista - UNESP",
                "license":[
                    {
                        "type":"CC by-nc",
                        "title":"CC by-nc"
                    }
                ],
                "language":[
                    "English"
                ],
                "title":"Journal of Venomous Animals and Toxins including Tropical Diseases",
                "country":"BR",
                "number":"1",
                "volume":"3"
            },
            "author":[
                {
                    "name":"S. STANILOVA"
                },
                {
                    "name":"Z. ZHELEV"
                },
                {
                    "name":"I. MATEV"
                }
            ],
            "link":[
                {
                    "url":"http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0104-79301997000100005",
                    "type":"fulltext"
                }
            ],
            "year":"1997",
            "keywords":[
                "Anti-complement",
                "venom",
                "Vipera ammodytes"
            ],
            "identifier":[
                {
                    "type":"doi",
                    "id":"10.1590/S0104-79301997000100005"
                },
                {
                    "type":"pissn",
                    "id":"0104-7930"
                },
                {
                    "type":"eissn",
                    "id":"1678-4936"
                }
            ],
            "abstract":"The present study demonstrates that Vipera ammodytes venom is a potent inhibitor of human complement activation. Crude venom was fractionated by ion-exchange chromatography and each fraction was then tested for proteolytic, hemolytic and anti-complement activities. Three peaks decreased the hemolytic activity of human serum when measured using assays for the classical and alternative pathways of activation. One of the anti-complement peaks contains a protease capable of cleaving the C3 component of complement.",
            "subject":[
                {
                    "code":"RC955-962",
                    "term":"Arctic medicine. Tropical medicine",
                    "scheme":"LCC"
                },
                {
                    "code":"RC952-1245",
                    "term":"Special situations and conditions",
                    "scheme":"LCC"
                },
                {
                    "code":"RC31-1245",
                    "term":"Internal medicine",
                    "scheme":"LCC"
                },
                {
                    "code":"R",
                    "term":"Medicine",
                    "scheme":"LCC"
                },
                {
                    "term":"Internal medicine",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Medicine (General)",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Health Sciences",
                    "scheme":"DOAJ"
                },
                {
                    "code":"QL1-991",
                    "term":"Zoology",
                    "scheme":"LCC"
                },
                {
                    "code":"Q",
                    "term":"Science",
                    "scheme":"LCC"
                },
                {
                    "term":"Zoology",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Biology",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Biology and Life Sciences",
                    "scheme":"DOAJ"
                },
                {
                    "scheme":"LCC",
                    "term":"Arctic medicine. Tropical medicine",
                    "code":"RC955-962"
                },
                {
                    "scheme":"LCC",
                    "term":"Special situations and conditions",
                    "code":"RC952-1245"
                },
                {
                    "scheme":"LCC",
                    "term":"Internal medicine",
                    "code":"RC31-1245"
                },
                {
                    "scheme":"LCC",
                    "term":"Medicine",
                    "code":"R"
                },
                {
                    "term":"Internal medicine",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Medicine (General)",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Health Sciences",
                    "scheme":"DOAJ"
                },
                {
                    "scheme":"LCC",
                    "term":"Zoology",
                    "code":"QL1-991"
                },
                {
                    "scheme":"LCC",
                    "term":"Science",
                    "code":"Q"
                },
                {
                    "term":"Zoology",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Biology",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Biology and Life Sciences",
                    "scheme":"DOAJ"
                },
                {
                    "code":"RC955-962",
                    "term":"Arctic medicine. Tropical medicine",
                    "scheme":"LCC"
                },
                {
                    "code":"RC952-1245",
                    "term":"Special situations and conditions",
                    "scheme":"LCC"
                },
                {
                    "code":"RC31-1245",
                    "term":"Internal medicine",
                    "scheme":"LCC"
                },
                {
                    "code":"R",
                    "term":"Medicine",
                    "scheme":"LCC"
                },
                {
                    "term":"Internal medicine",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Medicine (General)",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Health Sciences",
                    "scheme":"DOAJ"
                },
                {
                    "code":"QL1-991",
                    "term":"Zoology",
                    "scheme":"LCC"
                },
                {
                    "code":"Q",
                    "term":"Science",
                    "scheme":"LCC"
                },
                {
                    "term":"Zoology",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Biology",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Biology and Life Sciences",
                    "scheme":"DOAJ"
                },
                {
                    "scheme":"LCC",
                    "term":"Arctic medicine. Tropical medicine",
                    "code":"RC955-962"
                },
                {
                    "scheme":"LCC",
                    "term":"Special situations and conditions",
                    "code":"RC952-1245"
                },
                {
                    "scheme":"LCC",
                    "term":"Internal medicine",
                    "code":"RC31-1245"
                },
                {
                    "scheme":"LCC",
                    "term":"Medicine",
                    "code":"R"
                },
                {
                    "term":"Internal medicine",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Medicine (General)",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Health Sciences",
                    "scheme":"DOAJ"
                },
                {
                    "scheme":"LCC",
                    "term":"Zoology",
                    "code":"QL1-991"
                },
                {
                    "scheme":"LCC",
                    "term":"Science",
                    "code":"Q"
                },
                {
                    "term":"Zoology",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Biology",
                    "scheme":"DOAJ"
                },
                {
                    "term":"Biology and Life Sciences",
                    "scheme":"DOAJ"
                },
                {
                    "code":"RC955-962",
                    "term":"Arctic medicine. Tropical medicine",
                    "scheme":"LCC"
                },
                {
                    "code":"RC952-1245",
                    "term":"Special situations and conditions",
                    "scheme":"LCC"
                },
                {
                    "code":"RC31-1245",
                    "term":"Internal medicine",
                    "scheme":"LCC"
                },
                {
                    "code":"R",
                    "term":"Medicine",
                    "scheme":"LCC"
                },
                {
                    "code":"QL1-991",
                    "term":"Zoology",
                    "scheme":"LCC"
                },
                {
                    "code":"Q",
                    "term":"Science",
                    "scheme":"LCC"
                },
                {
                    "scheme":"LCC",
                    "term":"Arctic medicine. Tropical medicine",
                    "code":"RC955-962"
                },
                {
                    "scheme":"LCC",
                    "term":"Special situations and conditions",
                    "code":"RC952-1245"
                },
                {
                    "scheme":"LCC",
                    "term":"Internal medicine",
                    "code":"RC31-1245"
                },
                {
                    "scheme":"LCC",
                    "term":"Medicine",
                    "code":"R"
                },
                {
                    "scheme":"LCC",
                    "term":"Zoology",
                    "code":"QL1-991"
                },
                {
                    "scheme":"LCC",
                    "term":"Science",
                    "code":"Q"
                }
            ]
        }
    }
}
dommitchell commented 8 years ago

I was also reminded of the metadata migration that you carried out for me where we standardised on the licenses. #241 #242 I think this may be related.

richard-jones commented 8 years ago

I've put in a PR which might well fix this...

dommitchell commented 8 years ago

@richard-jones did this go live?

Steven-Eardley commented 8 years ago

Assuming @richard-jones meant commit https://github.com/DOAJ/doaj/commit/0165de1e872a9ca333c37b54e4f483b6159ac992, yes, his changes are live. Are you still seeing the 500 error?

dommitchell commented 8 years ago

Yep, no change: https://doaj.org/article/000c38480c3d4d1383444cf8b24bc85d

emanuil-tolev commented 8 years ago

Error is (still)

2016-01-04 14:27:41,565 - portality.core - ERROR - Exception on /article/000c38480c3d4d1383444cf8b24bc85d [GET]
Traceback (most recent call last):
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/flask/app.py", line 1687, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/flask/app.py", line 1360, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/flask/app.py", line 1358, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/flask/app.py", line 1344, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/cloo/repl/production/doaj/src/doaj/portality/view/doaj.py", line 278, in article_page
    return render_template('doaj/article.html', article=article, journal=journal)
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/flask/templating.py", line 125, in render_template
    context, ctx.app)
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/flask/templating.py", line 107, in _render
    rv = template.render(context)
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/home/cloo/repl/production/doaj/local/lib/python2.7/site-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/cloo/repl/production/doaj/src/doaj/portality/templates/doaj/article.html", line 1, in top-level template code
    {% extends "base.html" %}
  File "/home/cloo/repl/production/doaj/src/doaj/portality/templates/base.html", line 21, in top-level template code
    {% block extra_meta_tags %}{% endblock %}
  File "/home/cloo/repl/production/doaj/src/doaj/portality/templates/doaj/article.html", line 18, in block "extra_meta_tags"
    {% set jbib = journal.bibjson() %}
UndefinedError: 'None' has no attribute 'bibjson'

I believe in this case the problem might be that despite @richard-jones's relevant fix, the data on the article record is simply out of date.

The article references a journal that does not exist - see this extract from the article record I pasted above:

"issn":[
    "1678-4936",
    "0104-7930"
]

Neither of these ISSNs exist in the catalogue, not even in the administrative site search. What likely happened was that either:

I believe the correct journal (judging by the title, which we thankfully record on the article itself) is https://doaj.org/toc/1678-9199 . There seem to be 253 articles affected by the non-existent ISSNs.

Shall we simply mass-update them? So we take all articles with those ISSNs and that journal title and replace the ISSNs in them with "1678-9199" which seems to be the right ISSN. It'll take .. probably about 30-60 min to formulate a query to get them, update each and save, and deal with any potential problems during that process.

Steven-Eardley commented 8 years ago

Shall we simply mass-update them? So we take all articles with those ISSNs and that journal title and replace the ISSNs in them with "1678-9199" which seems to be the right ISSN.

Now I see that, I recall the work is already scheduled - I think this is part of my task for #941, which I'm doing imminently.

richard-jones commented 8 years ago

Added this to the harvester milestone, as it'll be fixed when the journal info script is fixed.

dommitchell commented 8 years ago

@Steven-Eardley can this one be closed now?

Steven-Eardley commented 8 years ago

I'm still seeing the issue after the release, and running the script that should fix it. Needs to be investigated some more.

richard-jones commented 7 years ago

These articles appear to be gone now, the link at the top leads to no results, so I'm closing this off.

emanuil-tolev commented 7 years ago

Are we syncing the journal info present in article records now? (I think we are, article_cleanup_sync type jobs) If so, that'll fix it anyway.