Reference importer not working; seems to fail in crossref.py "No JSON object could be decode"

soto97 commented 4 years ago

As of today, the reference importer workflow for Alfred is not working for me. I type 'ref' followed by search terms and then I get nothing. Below I have the debug log output from Alfred. You will see me try to different searches. The first one, I just copied and pasted in the query. The second one, I typed the query terms, so you see the queuing working correctly. In both cases the workflow fails and the error message says, ValueError: No JSON object could be decoded.

Does anyone have any idea what is going on and how to fix it?

Note: I am running on Catalina and I have the most recent version of Reference Importer installed on Alfred 4.1.1.

[13:24:46.560] Logging Started...
[13:24:51.728] Reference Importer[Script Filter] Queuing argument 'A Tutorial on Lateral Boundary Conditions as a Basic and Potentially Serious Limitation to Regional Numerical Weather Prediction'
[13:24:52.610] Reference Importer[Script Filter] Script with argv '(null)' finished
[13:24:52.613] ERROR: Reference Importer[Script Filter] Code 1: Traceback (most recent call last):
  File "crossref.py", line 36, in <module>
    for j in r.json():
  File "/Users/asoto/Library/Application Support/Alfred/Alfred.alfredpreferences/workflows/user.workflow.621446A5-1079-4155-9F7E-7A1416FC6D41/requests/models.py", line 638, in json
    return json.loads(self.text or self.content, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
[13:26:03.860] Reference Importer[Script Filter] Queuing argument 'm'
[13:26:03.993] Reference Importer[Script Filter] Queuing argument 'ma'
[13:26:04.149] Reference Importer[Script Filter] Queuing argument 'mar'
[13:26:04.307] Reference Importer[Script Filter] Queuing argument 'mart'
[13:26:04.464] Reference Importer[Script Filter] Queuing argument 'marti'
[13:26:04.647] Reference Importer[Script Filter] Queuing argument 'martia'
[13:26:04.895] Reference Importer[Script Filter] Queuing argument 'martian'
[13:26:04.904] Reference Importer[Script Filter] Script with argv '(null)' finished
[13:26:04.906] ERROR: Reference Importer[Script Filter] Code 1: Traceback (most recent call last):
  File "crossref.py", line 36, in <module>
    for j in r.json():
  File "/Users/asoto/Library/Application Support/Alfred/Alfred.alfredpreferences/workflows/user.workflow.621446A5-1079-4155-9F7E-7A1416FC6D41/requests/models.py", line 638, in json
    return json.loads(self.text or self.content, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
[13:26:05.049] Reference Importer[Script Filter] Queuing argument 'martian '
[13:26:05.297] Reference Importer[Script Filter] Queuing argument 'martian a'
[13:26:05.431] Reference Importer[Script Filter] Queuing argument 'martian at'
[13:26:05.567] Reference Importer[Script Filter] Queuing argument 'martian atm'
[13:26:05.702] Reference Importer[Script Filter] Queuing argument 'martian atmo'
[13:26:05.719] Reference Importer[Script Filter] Script with argv '(null)' finished
[13:26:05.719] ERROR: Reference Importer[Script Filter] Code 1: Traceback (most recent call last):
  File "crossref.py", line 36, in <module>
    for j in r.json():
  File "/Users/asoto/Library/Application Support/Alfred/Alfred.alfredpreferences/workflows/user.workflow.621446A5-1079-4155-9F7E-7A1416FC6D41/requests/models.py", line 638, in json
    return json.loads(self.text or self.content, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
[13:26:05.904] Reference Importer[Script Filter] Queuing argument 'martian atmos'
[13:26:05.994] Reference Importer[Script Filter] Queuing argument 'martian atmosp'
[13:26:06.265] Reference Importer[Script Filter] Queuing argument 'martian atmosph'
[13:26:06.283] Reference Importer[Script Filter] Script with argv '(null)' finished
[13:26:06.284] ERROR: Reference Importer[Script Filter] Code 1: Traceback (most recent call last):
  File "crossref.py", line 36, in <module>
    for j in r.json():
  File "/Users/asoto/Library/Application Support/Alfred/Alfred.alfredpreferences/workflows/user.workflow.621446A5-1079-4155-9F7E-7A1416FC6D41/requests/models.py", line 638, in json
    return json.loads(self.text or self.content, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
[13:26:06.377] Reference Importer[Script Filter] Queuing argument 'martian atmosphe'
[13:26:06.534] Reference Importer[Script Filter] Queuing argument 'martian atmospher'
[13:26:06.715] Reference Importer[Script Filter] Queuing argument 'martian atmospheri'
[13:26:07.045] Reference Importer[Script Filter] Script with argv '(null)' finished
[13:26:07.070] ERROR: Reference Importer[Script Filter] Code 1: Traceback (most recent call last):
  File "crossref.py", line 36, in <module>
    for j in r.json():
  File "/Users/asoto/Library/Application Support/Alfred/Alfred.alfredpreferences/workflows/user.workflow.621446A5-1079-4155-9F7E-7A1416FC6D41/requests/models.py", line 638, in json
    return json.loads(self.text or self.content, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
[13:26:07.459] Reference Importer[Script Filter] Queuing argument 'martian atmospheric'
[13:26:07.751] Reference Importer[Script Filter] Queuing argument 'martian atmospheric '
[13:26:07.841] Reference Importer[Script Filter] Script with argv '(null)' finished
[13:26:07.848] ERROR: Reference Importer[Script Filter] Code 1: Traceback (most recent call last):
  File "crossref.py", line 36, in <module>
    for j in r.json():
  File "/Users/asoto/Library/Application Support/Alfred/Alfred.alfredpreferences/workflows/user.workflow.621446A5-1079-4155-9F7E-7A1416FC6D41/requests/models.py", line 638, in json
    return json.loads(self.text or self.content, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
[13:26:07.999] Reference Importer[Script Filter] Queuing argument 'martian atmospheric c'
[13:26:08.086] Reference Importer[Script Filter] Queuing argument 'martian atmospheric co'
[13:26:08.312] Reference Importer[Script Filter] Queuing argument 'martian atmospheric col'
[13:26:08.469] Reference Importer[Script Filter] Queuing argument 'martian atmospheric coll'
[13:26:08.524] Reference Importer[Script Filter] Script with argv '(null)' finished
[13:26:08.533] ERROR: Reference Importer[Script Filter] Code 1: Traceback (most recent call last):
  File "crossref.py", line 36, in <module>
    for j in r.json():
  File "/Users/asoto/Library/Application Support/Alfred/Alfred.alfredpreferences/workflows/user.workflow.621446A5-1079-4155-9F7E-7A1416FC6D41/requests/models.py", line 638, in json
    return json.loads(self.text or self.content, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
[13:26:08.582] Reference Importer[Script Filter] Queuing argument 'martian atmospheric colla'
[13:26:08.694] Reference Importer[Script Filter] Queuing argument 'martian atmospheric collap'
[13:26:08.806] Reference Importer[Script Filter] Queuing argument 'martian atmospheric collaps'
[13:26:08.989] Reference Importer[Script Filter] Queuing argument 'martian atmospheric collapse'
[13:26:09.167] Reference Importer[Script Filter] Script with argv '(null)' finished
[13:26:09.181] ERROR: Reference Importer[Script Filter] Code 1: Traceback (most recent call last):
  File "crossref.py", line 36, in <module>
    for j in r.json():
  File "/Users/asoto/Library/Application Support/Alfred/Alfred.alfredpreferences/workflows/user.workflow.621446A5-1079-4155-9F7E-7A1416FC6D41/requests/models.py", line 638, in json
    return json.loads(self.text or self.content, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
[13:26:09.923] Reference Importer[Script Filter] Script with argv '(null)' finished
[13:26:09.943] ERROR: Reference Importer[Script Filter] Code 1: Traceback (most recent call last):
  File "crossref.py", line 36, in <module>
    for j in r.json():
  File "/Users/asoto/Library/Application Support/Alfred/Alfred.alfredpreferences/workflows/user.workflow.621446A5-1079-4155-9F7E-7A1416FC6D41/requests/models.py", line 638, in json
    return json.loads(self.text or self.content, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

andrewning commented 4 years ago

yeah I'm getting the same error. I'm not sure offhand. Occasionally the CrossRef API goes down. I'll wait a few days and see if it resolves itself first.

tobiasgerstenberg commented 4 years ago

the reference importer is not working anymore for me either (on alfred 4.2.1 [1187])

soto97 commented 3 years ago

Digging deeper into this, when crossref.py executes this line:

r = requests.get('http://search.crossref.org/dois', params=params)

the result cannot be handled by the json function in models.py.

To test things, right after the value r is populated in crossref.py, but before the for loop where the values of r are split up, I just put this print statement:

print('r = ', r.json['doi'])

Basically, I just wanted to know what the DOI was for the query. I got this error message:

[15:17:11.683] Reference Importer[Script Filter] Queuing argument 'The vertical distribution of Martian aerosol
particle size'
[15:17:12.249] Reference Importer[Script Filter] Script with argv '(null)' finished
[15:17:12.251] ERROR: Reference Importer[Script Filter] Code 1: Traceback (most recent call last):
  File "crossref.py", line 34, in <module>
    print('r = ', r.json['doi'])
TypeError: 'instancemethod' object has no attribute '__getitem__'

Maybe I am doing something wrong, or maybe there's a miscommunication with CrossRef and they are not sending a properly formatted JSON output? Or did they change that formatting. There are packages in the models.py code that I am not familiar with nor do I know where they are coming from, so I have hit another dead end with this (at least for me).

Note: running the command ref doi_query!! does work, so I can at least get bibtex entries by entering DOIs. Useful for recent papers, at least.

@andrewning do you have the same problem when you use crossref?

Thanks!

emraher commented 3 years ago

I'm not sure but I think API has changed. Using http://search.crossref.org/dois gives IP block errors.

When I change lines

params = {'q': query, 'rows': '10'} r = requests.get('http://search.crossref.org/dois', params=params)

to

params = {'query': query, 'rows': '10'} r = requests.get('https://api.crossref.org/works?', params=params)

I get json response from API.

I couldn't fix the rest though since I'm not very good at Python.

EDIT: I think I fixed it but probably someone can write a better code for this. This is what I have in crossref.py file

    # Use crossref metadata search (beta) to get the DOI
    params = {'query': query, 'rows': '10'}
    r = requests.get('http://api.crossref.org/works', params=params)
    data = r.json()
    message = list(data.values())[2]
    items = list(message.values())[0]

    # write results in XML format for Alfred
    results = []
    for j in items:
        doi = j['DOI']
        title = j['title'][0]
        subtitle = j['container-title'][0]
        # info = j['fullCitation']
        # entries = info.split('\'')
        # subtitle = entries[0]
        # if len(entries) > 1:
            # subtitle += (''.join(entries[2:])[2:])

        # strip out html tag for italic
        # subtitle = subtitle.replace('<i>', '')
        # subtitle = subtitle.replace('</i>', '')
        results.append(alfred.Item(title=title,
                                   subtitle=subtitle,
                                   attributes={'uid': doi, 'arg': doi},
                                   icon='crossref.png'))

andrewning commented 3 years ago

yes, I've been aware of the problem and API change for a few weeks, just haven't had time to rework things. Hopefully soon. Looks like @emraher has a good start.

soto97 commented 3 years ago

I started with the potential fix that @emraher provided above. That works great.

However, @emraher set the subtitle to the journal name (subtitle = j['container-title'][0]). Although this is useful, I feel like Reference Importer used to put the author names in that field, which I find more useful for making sure I have the correct reference. When I tried to do that, however, it didn't go well.

In the current Crossref API, the author field is a nested dictionary. The author field looks like this in the raw output: 'author': [{'affiliation': [ ], 'given': [ ], 'family': [ ], 'sequence': [ ]}] The dictionary includes a number of keys, but only one is required, the 'family' key. Therefore, I replaced the subtitle call with:

subtitle = j['author'][0]['family'][:]

This works, but only partially. When I provided only one word as the search term, for example, ref mar, I would get a list of papers about Mars with the author given names in the subtitle field. However, once I tried multiple words, for example, ref mars atmosphere, the query failed. I am stuck now, so I share this with everyone to see if someone else has a solution.

I encourage @emraher to submit their fix as a pull request, because it works and others may want it.

I also ask that if there is a way to put the author info in the subtitle field, I think that would really make the results more useful.

Thanks.

emraher commented 3 years ago

The updated workflow is in my fork's dev branch. Here is the link for the Alfred Workflow file.

I tried to fix the reference importer using sckott/habanero for Crossref API and deanishe/alfred-workflow.
I tested with a couple of articles and it seems it is working now.
I have added Author and Journal Info in the subtitle as you can see below.
Google Books seems to work but Google Scholar has problems.
I hope someone with more Python knowledge writes a better code soon.

tobiasgerstenberg commented 3 years ago

thanks for taking a crack at this! with this update, book works for me but neither does ref nor gsref.

emraher commented 3 years ago

thanks for taking a crack at this! with this update, book works for me but neither does ref nor gsref.

@tobiasgerstenberg have you tried this? What was the error? Also see https://github.com/andrewning/alfred-workflows-scientific/issues/6#issuecomment-357038107

tobiasgerstenberg commented 3 years ago

yes, i've tried this one. i don't get an error message when trying to use ref. the message "searching for matching articles" is displayed for a bit and then alfred just switches to searching via google.

having played around with it for a bit now, it sometimes does work (but not reliably); and it doesn't seem to be dependent on the search query. the same query sometimes works and sometimes doesn't.

emraher commented 3 years ago

@tobiasgerstenberg can you check the log messages in Alfred's debugging mode?

tobiasgerstenberg commented 3 years ago

it's working reliably for me now (without me having changed anything). thanks a lot for the update!

soto97 commented 3 years ago

@emraher Thanks for the updates that you made. I really like some of the new tweaks.

I did run into one problem: With the version of the workflow in your repository, I got an error for some crossref searches. Here's the error message from the Alfred debugger:

workflow.py:2080 ERROR    list indices must be integers, not str
Traceback (most recent call last):
  File ".../workflow/workflow.py", line 2073, in run
    func(self)
  File "crossref.py", line 60, in main
    journal = item['institution']['name']
TypeError: list indices must be integers, not str

(Note: I removed some details of the file path (where I wrote "...") for workflow.py because they are not important and are specific to my machine.)

There was an easy fix for this: I just changed line 60 in crossref.py (the line called out in the error message above) to the following:

                journal = item['institution'][0]

Basically, I replaced 'name' with 0. So far this seems to work great, with the matches showing paper title, author names and the citation source (i.e., the journal). Hopefully, the edit I made won't cause some other problem down the line.

Again, thanks for sharing this fix to respond to the changes in the CrossRef API.

andrewning / alfred-workflows-scientific

Reference importer not working; seems to fail in crossref.py "No JSON object could be decode" #17

The updated workflow is in my fork's dev branch. Here is the link for the Alfred Workflow file.