search

Gertje823 / Vinted-Scraper

This is a tool to scrape/download images and data from Vinted & Depop using the API and stores the data in a SQLite database.
GNU General Public License v3.0
91 stars 21 forks source link

multiple user IDs at once #7

Closed exuded closed 2 years ago

exuded commented 2 years ago

Hi :) is it possible to input multiple user IDs at once? And how can I get the script to run every 5 mins for example?

Gertje823 commented 2 years ago

Yes, you can enter 1 userid per line in your users.txt. example: 12345
12346
12347

If you want to run the script every 5 minutes you can run it as a cronjob or if you are on windows you could run it through task scheduler.

exuded commented 2 years ago

I added 1 User ID per line, but when I execute the script, it checks for the first ID in line 1 of users.txt. After that it does not continue with the second User ID, but gives this error log:

Traceback (most recent call last): File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main\Vinted-Scraper-main\scraper.py", line 343, in download_vinted_data(userids, session, params) File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main\Vinted-Scraper-main\scraper.py", line 64, in download_vinted_data r = s.get(url, params=params) File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main\Vinted-Scraper-main\venv\lib\site-packages\requests\sessions.py", line 543, in get return self.request('GET', url, **kwargs) File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main\Vinted-Scraper-main\venv\lib\site-packages\requests\sessions.py", line 516, in request prep = self.prepare_request(req) File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main\Vinted-Scraper-main\venv\lib\site-packages\requests\sessions.py", line 449, in prepare_request p.prepare( File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main\Vinted-Scraper-main\venv\lib\site-packages\requests\models.py", line 314, in prepare self.prepare_url(url, params) File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main\Vinted-Scraper-main\venv\lib\site-packages\requests\models.py", line 432, in prepare_url enc_params = self._encode_params(params) File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main\Vinted-Scraper-main\venv\lib\site-packages\requests\models.py", line 97, in _encode_params for k, vs in to_key_val_list(data): ValueError: too many values to unpack (expected 2)

Process finished with exit code 1

Gertje823 commented 2 years ago

I think the format of the request parameters caused this error. Could you try again with the latest version? It should be fixed.

exuded commented 2 years ago

I just putted in 5 IDs and got this error:

Traceback (most recent call last): File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main V2\Vinted-Scraper-main\scraper.py", line 339, in download_vinted_data(userids, session, params) File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main V2\Vinted-Scraper-main\scraper.py", line 60, in download_vinted_data r = s.get(url, params=params) File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main\Vinted-Scraper-main\venv\lib\site-packages\requests\sessions.py", line 543, in get return self.request('GET', url, **kwargs) File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main\Vinted-Scraper-main\venv\lib\site-packages\requests\sessions.py", line 516, in request prep = self.prepare_request(req) File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main\Vinted-Scraper-main\venv\lib\site-packages\requests\sessions.py", line 449, in prepare_request p.prepare( File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main\Vinted-Scraper-main\venv\lib\site-packages\requests\models.py", line 314, in prepare self.prepare_url(url, params) File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main\Vinted-Scraper-main\venv\lib\site-packages\requests\models.py", line 432, in prepare_url enc_params = self._encode_params(params) File "C:\Users\Synology-vm-1\Downloads\Vinted-Scraper-main\Vinted-Scraper-main\venv\lib\site-packages\requests\models.py", line 97, in _encode_params for k, vs in to_key_val_list(data): TypeError: cannot unpack non-iterable int object

Gertje823 commented 2 years ago

Weird.. it only happens at the second user. Turns out it has something to do with the request parameters. Apparently Vinted changed something and the parameter is no longer required so I removed it and now it is working for me.