This PR introduces improvements to our Twitter scraping authentication process to make it more robust and less detectable as automated behavior.
Changes
In config.go:
Introduced RandomSleep() function to add variability in request timing
Set sleep duration range from 500ms to 2s
Initialized a package-level random number generator for thread-safety
In auth.go:
Updated NewScraper() to use RandomSleep() before and after login attempts
Improved error handling and logging in the login process
Rationale
These changes aim to make our scraping behavior more human-like and reduce the risk of being detected as a bot or having our accounts suspended. By introducing random delays between actions, we mimic natural user behavior more closely.
Testing
Tested login process with multiple accounts
Verified that random sleep durations are being applied correctly
Ensured that error handling and logging are working as expected
References
auth.go:
startLine: 9
endLine: 35
config.go:
startLine: 10
endLine: 32
Checklist
[x] Code follows the project's coding standards
[x] Comments have been added where necessary
[x] All tests pass
[x] Documentation has been updated (if applicable)
Description
This PR introduces improvements to our Twitter scraping authentication process to make it more robust and less detectable as automated behavior.
Changes
In
config.go
:RandomSleep()
function to add variability in request timingIn
auth.go
:NewScraper()
to useRandomSleep()
before and after login attemptsRationale
These changes aim to make our scraping behavior more human-like and reduce the risk of being detected as a bot or having our accounts suspended. By introducing random delays between actions, we mimic natural user behavior more closely.
Testing
References
auth.go
:config.go
:Checklist