masa-finance / masa-oracle

Masa Oracle: Decentralized Data Protocol 🌐
https://developers.masa.ai/docs/masa-protocol/welcome
MIT License
24 stars 19 forks source link

feat(twitter): Implement random sleep and improve login process #601

Closed teslashibe closed 1 month ago

teslashibe commented 1 month ago

Description

This PR introduces improvements to our Twitter scraping authentication process to make it more robust and less detectable as automated behavior.

Changes

  1. In config.go:

    • Introduced RandomSleep() function to add variability in request timing
    • Set sleep duration range from 500ms to 2s
    • Initialized a package-level random number generator for thread-safety
  2. In auth.go:

    • Updated NewScraper() to use RandomSleep() before and after login attempts
    • Improved error handling and logging in the login process

Rationale

These changes aim to make our scraping behavior more human-like and reduce the risk of being detected as a bot or having our accounts suspended. By introducing random delays between actions, we mimic natural user behavior more closely.

Testing

References

Checklist