search

sfu-db / APIConnectors

A curated list of example code to collect data from Web APIs using DataPrep.Connector.
https://github.com/sfu-db/dataprep#connector
34 stars 25 forks source link

Multiple variables in same request #137

Open Lakshay-sethi opened 3 years ago

Lakshay-sethi commented 3 years ago

Describe the bug First of all , this is not a critical issue.

Basically, while reviewing I faced the following issue. I saw that the developer was using a rudimentary form of getting the data, upon further examination, it was brought to my attention that system throws a assertion error when the config file is changed, as shown below.

image

image

To Reproduce Steps to reproduce the behavior:

from dataprep.connector import connect
# You can get ”app_key“ by following https://www.themuse.com/developers/api/v2/apps
dc = connect('themuse', _auth={'access_token': app_key})
df = await dc.query('jobs', page=1, category='Data Science', location='Vancouver, Canada')
df[['id', 'name', 'company', 'locations', 'levels', 'publication_date']]
Desired behavior id name company locations levels publication_date
0 5126286 Senior Data Scientist Discord ['Flexible / Remote'] ['Senior Level'] 2021-03-15T11:10:24Z
1 5543215 Data Scientist-AI/ML (Remote) Dell Technologies ['Chicago, IL', 'Flexible /...] ['Mid Level'] 2021-04-02T11:45:57Z
2 4959228 Senior Data Scientist Humana ['Flexible / Remote'] ['Senior Level'] 2021-01-05T11:28:23.814281Z
Current behavior id name company locations levels publication_date
0 5126286 Senior Data Scientist Discord [{'name': 'Flexible / Remote] [{'name': 'Senior Level', 'short_name': 'senio... 2021-03-15T11:10:24Z
1 5543215 Data Scientist-AI/ML (Remote) Dell Technologies [{'name': 'Chicago, IL'}, {'name': 'Flexible /... [{'name': 'Mid Level', 'short_name': 'mid'}] 2021-04-02T11:45:57Z
2 4959228 Senior Data Scientist Humana [{'name': 'Flexible / Remote'}] [{'name': 'Senior Level', 'short_name': 'senio... 2021-01-05T11:28:23.814281Z

Additional context same issue found in airplanes api

nick-zrymiak commented 3 years ago

Interesting!

"system throws a assertion error when the config file is changed, as shown below."

What is being changed?

Lakshay-sethi commented 3 years ago

When I change the code to what is shown in picture

nick-zrymiak commented 3 years ago

Hmm but what did it look like before?

Lakshay-sethi commented 3 years ago

image

peiwangdb commented 3 years ago

Interesting! @Lakshay-sethi it seems the issue is that our config file template has very strong assumptions and can not tolerate variance.