NoahSaso / merge-psychonautwiki-tripsit-data

Script to merge data from PsychonautWiki API + pages and TripSit factsheets into one standardized format
MIT License
11 stars 8 forks source link

TypeError: 'NoneType' object is not iterable #3

Open pyt0xic opened 2 years ago

pyt0xic commented 2 years ago

It worked initially but now it isnt, weird, here is the error.

Traceback (most recent call last):
  File "/home/nic/Coding/pablo-bot/ts_pn_data/getData.py", line 184, in <module>
    for idx, substance in enumerate(pw_substance_urls_data):
TypeError: 'NoneType' object is not iterable

I will keep trying to fix it

NoahSaso commented 2 years ago

Looks like there's some internal server error with the PsychonautWiki API :/

https://api.psychonautwiki.org/ Run this query:

    {
        substances(limit: 1, offset: 251) {
            name
            url
        }
    }

Seems like the 251st substance (and some other ones, but I just tracked down the first one to cause the error) have some data issues going on.

pyt0xic commented 2 years ago

Yeah I worked it just now lol. I think maybe its too do with calling with too high of a limit because it seems to break at even 200 at times. It worked earlier with 1000 so I am quite confused

I think I found a work around using a while loop and gradually increasing the offset, I will share when I've finished testing

pyt0xic commented 2 years ago

I just tested that query, I see what you mean, I guess We will just have call each one by one and put a check for Null returns...

I will try implement something.

pyt0xic commented 2 years ago

A very ugly but working solution

if not len(pw_substance_data):
    offset = 0
    pw_substance_urls_query = (
        f"{{substances(limit: 250 offset: {offset}) {{name url}}}}"
    )

    pw_substance_urls_data = ps_client.execute(query=pw_substance_urls_query,)["data"][
        "substances"
    ]

    offset = 252
    while offset <= 340:
        pw_substance_urls_query = (
            f"{{substances(limit: 1 offset: {offset}) {{name url}}}}"
        )
        offset += 1
        temp_data = ps_client.execute(query=pw_substance_urls_query,)["data"][
            "substances"
        ]
        print(temp_data)
        if temp_data is None:
            continue
        pw_substance_urls_data.extend(temp_data)

Not sure if you know of a better way to do this?