pocesar / actor-twitter-scraper

Scrape any Twitter user profile. Extract tweets, retweets, replies, favorites, and conversation threads with no Twitter API limits
https://apify.com/vdrmota/twitter-scraper
Apache License 2.0
12 stars 11 forks source link

Topic scraping not working? #9

Closed jnorris441 closed 3 years ago

jnorris441 commented 3 years ago

Put in a topic URL to see if any tweets would come back. Is this the correct usage? It returns 0 items unfortunately.

Input:

{
  "tweetsDesired": 100,
  "startUrls": [
    {
      "url": "https://twitter.com/i/topics/1010220507482865665"
    }
  ],
  "addUserInfo": true,
  "mode": "own",
  "proxyConfig": {
    "useApifyProxy": true
  },
  "extendOutputFunction": "async ({ data, item, page, request, customData, Apify }) => {\n\t      return item;\n\t    }",
  "extendScraperFunction": "async ({ page, request, addSearch, addProfile, _, addThread, addEvent, customData, Apify, signal, label }) => {\n\t     \n\t    }",
  "customData": {},
  "handlePageTimeoutSecs": 5000,
  "initialCookies": [],
  "handle": [],
  "stealth": false
}

Run log:

2021-05-09T03:27:06.924Z ACTOR: Pulling Docker image from repository.
2021-05-09T03:27:06.992Z ACTOR: Creating Docker container.
2021-05-09T03:27:07.124Z ACTOR: Starting Docker container.
2021-05-09T03:27:11.889Z INFO  System info {"apifyVersion":"1.1.2","apifyClientVersion":"1.1.0","osType":"Linux","nodeVersion":"v14.16.1"}
2021-05-09T03:27:12.322Z INFO  Limiting tweet counts to 100...
2021-05-09T03:27:14.746Z INFO  Starting scraper
2021-05-09T03:27:14.811Z INFO  PuppeteerCrawler:AutoscaledPool: state {"currentConcurrency":0,"desiredConcurrency":1,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":null},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.6,"actualRatio":null},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":null},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":null}}}
2021-05-09T03:27:41.396Z WARN  Scrolling seems to have stopped and data was not received after 20s
2021-05-09T03:27:41.603Z INFO  Finished with https://twitter.com/i/topics/1010220507482865665
2021-05-09T03:27:41.605Z INFO  Extracted 0 tweets from https://twitter.com/i/topics/1010220507482865665
2021-05-09T03:27:41.792Z INFO  PuppeteerCrawler: All the requests from request list and/or request queue have been processed, the crawler will shut down.
2021-05-09T03:27:42.399Z INFO  PuppeteerCrawler: Final request statistics: {"requestsFinished":1,"requestsFailed":0,"retryHistogram":[1],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":26786,"requestsFinishedPerMinute":2,"requestsFailedPerMinute":0,"requestTotalDurationMillis":26786,"requestsTotal":1,"crawlerRuntimeMillis":27654}
2021-05-09T03:27:42.401Z INFO  All finished
pocesar commented 3 years ago

this is fixed in latest