jwyg
commented
3 years ago Though I'm still relatively new to 4CAT I've been exploring temporary fixes for removing things from worker queue and leaving this here in case useful for others...
If running 4CAT with Docker, one can explore the database using psql with the following:
docker exec -it db /bin/bash
psql fourcat fourcatTo list tables in the database:
\dWhich shows the following:
List of relations
Schema | Name | Type | Owner
--------+------------------+----------+---------
public | access_tokens | table | fourcat
public | datasets | table | fourcat
public | datasets_id_seq | sequence | fourcat
public | jobs | table | fourcat
public | jobs_id_seq | sequence | fourcat
public | metrics | table | fourcat
public | users | table | fourcat
public | users_favourites | table | fourcat
(8 rows)To show items in jobs:
SELECT * from jobs;Which lists jobs with the following structure
fourcat=# SELECT * from jobs;
id | jobtype | remote_id | details | timestamp | timestamp_after | timestamp_lastclaimed | timestamp_claimed | status | attempts | interval
----+----------------------+----------------------------------+---------+------------+-----------------+-----------------------+-------------------+--------+----------+----------Then to remove a specific job:
DELETE FROM jobs
WHERE id = [the id of the job that you would like to delete];This succeeds in removing jobs - but not sure whether this is advisable or may lead to problems or unexpected behaviours later on. Any advice on best practices for removing jobs would be much appreciated!
dale-wahl
stijn-uva
We have our 4CAT instance up and running thanks to the wonderful docker container (thanks for advice and support @stijn-uva)🎈 🥳
When testing and setting up data collections @lilianabounegru has found that API queries with a date range (i.e. beyond the default 30 day window) seem to hang.
This is the case even with relatively small queries (e.g. 10 items) and with relatively small date ranges (e.g. 1 day). These queries remain uncompleted for days with the little sand timer icon on the dataset page and with "twitterv2-search" items remaining in the worker queue, and we've yet to successfully complete a dataset with a date range.
Also it seems that as these queries are not completing, other queries are also stalled.
We are set up to collect Twitter data using API.v2 (Academic Track) and have no issues getting data without date parameters.
The API tokens/keys definitely in other contexts (e.g. with twarc), and no issues with getting data within window of previous 30 days. We also checked the same bearer token on OILab 4cat instance and datasets complete without issue.
So we were wondering: