search

datawhores / OF-Scraper

A completely revamped and redesigned fork, reimagined from scratch based on the original onlyfans-scraper
MIT License
654 stars 54 forks source link

Auth Loop When Trying to Download #426

Closed JSylvia007 closed 2 months ago

JSylvia007 commented 3 months ago

Describe the bug

Running both the latest release and the latest DEV version, I get in an auth loop, which invalidates my logins session, even on the web.

To Reproduce

Login using FF, use Cookie Helper to grab info, paste into OF-Scraper, try to download, see this message repeated multople times. Refresh the FF page, logged out.

REPEATED MESSAGE:

This info is only printed to console

{
  REDADCTED SESSION INFORMATION.
}
 Dynamic Rule: deviint

==============================================================
Double check to make sure the [x-bc,user-agent] info is correct

Double check to make sure the [sess, auth_id] info is correct

If 2fa is enabled double check that [auth_uid_] is set and not the same as auth_id

Double check to make sure dynamic rule is as desired
================================================

Expected behavior

I should be able to download.

Screenshots/Logs

Logs are required for any reports of any issue, unless the issue is related to setup of script Logs must be at least debug level

Hint: Please make sure that the log formatting is readable, use a paste site like Private Bin

Config

Using a standard config that's been working. Haven't changed defaults.

System Info

datawhores commented 3 months ago

Can you try the latest commit on the 3.10b branch I made some changes

JSylvia007 commented 3 months ago

Can you try the latest commit on the 3.10b branch I made some changes

Sure - What's the command to install from that branch?

I tried this pip install git+https://github.com/datawhores/OF-Scraper/tree/3.10b and that didn't work.

CucciPro commented 3 months ago

pip install git+https://github.com/datawhores/OF-Scraper.git@3.10b is what you want

JSylvia007 commented 3 months ago

pip install git+https://github.com/datawhores/OF-Scraper.git@3.10b is what you want

This worked, and it seems to be connecting, but man is it ungodly slow.

datawhores commented 3 months ago

I added a sleep line to test something it has been removed should be faster now

JSylvia007 commented 3 months ago

I added a sleep line to test something it has been removed should be faster now

I ran the same checkout line, but I don't think it made a difference. Still running really slowly.

datawhores commented 3 months ago

Try --no-cache maybe

JSylvia007 commented 3 months ago

Try --no-cache maybe

Negative unfortunately. And upon further inspection, while it LOOKS like it's doing something and looping through what I want it to download, it's not actually downloading anything.

datawhores commented 3 months ago

No I meant add --no-cache to the pip command

datawhores commented 3 months ago

Also try ofscraper -h

and ofscraper --media-id

And ofscraper --post-id And provide the output of thoae

JSylvia007 commented 3 months ago

Tried the new pip command... doesn't seem to make a difference. Still VERY slow.

Here is the output of those commands:

Note, running either --media-id or --post-id resulted in a crash / stacktrace. Do you still want that output?

(ofscraper) root@plex01:/opt/ofscraper# ofscraper -v
OF-Scraper, version 3.10.7
(ofscraper) root@plex01:/opt/ofscraper# ofscraper -h
Usage: OF-Scraper [OPTIONS] COMMAND [ARGS]...

  OF-Scraper Options

Program Options:
  Control the application's behavior with these settings
  -v, --version                   Show the version and exit.
  -cg, --config TEXT              Change location of config folder/file
  -r, --profile TEXT              Change which profile you want to use If not
                                  set then the config file is used Profiles are
                                  always within the config file parent directory

Logging Options:
  Settings for logging
  -l, --log [OFF|STATS|LOW|NORMAL|DEBUG|TRACE]
                                  Set log file level
  -dc, --discord [OFF|STATS|LOW|NORMAL|DEBUG|TRACE]
                                  Set discord log level
  -p, --output [PROMPT|STATS|LOW|NORMAL|DEBUG|TRACE]
                                  Set console output log level
  -nl, --no-live-screen, --no-live, --no-rich
                                  Turn off rich live display features

Download Options:
  Options for downloads and download performance
  -ar, --no-auto-resume           Cleanup temp .part files (removes resume
                                  ability)
  -db, --downloadbars             Show individual download progress bars
  -sd, --downloadsem, --downloadsems, --sems, --sem INTEGER
                                  Number of concurrent downloads per thread
  -dp, --downloadthreads, --threads, --thread, --downloadthreaddownloadthreads INTEGER
                                  Number of threads to use (minimum 1)

Media Filters Options:
  Options for controlling which media is downloaded
  -q, --quality [240|720|source]
  -mt, --mediatype MEDIATYPE_HELPER
                                  Filter by media type (Videos, Audios, Images)
  -sx, --size-max PARSE_SIZE      Filter out files larger than the given size
                                  (bytes or human-readable, e.g., 10mb)
  -sm, --size-min PARSE_SIZE      Filter out files smaller than the given size
                                  (bytes or human-readable, e.g., 10mb)
  -to, --protected-only           Restricts downloads to content that requires
                                  decryption.
  -no, --normal-only              Restricts downloads to content that does not
                                  require decryption.

Filename Modification Options:
  Options for controlling the output of the final filename after placeholders are replaced
  -g, --original                  Don't truncate long paths
  -tt, --text-type [word|letter]  set length based on word or letter
  -sr, --space-replacer TEXT      character to replace spaces with
  -tl, --textlength INTEGER       max length of text

Content Options:
          Define what posts to target (areas, filters) and actions to perform (like, unlike, download)
          Filter by type, date, label, size, and media type
  -o, --posts, --post POST_TYPE_HELPER
                                  Select areas for batch actions (comma or space
                                  separated). Options: HighLights, Archived,
                                  Messages, Timeline, Pinned, Stories,
                                  Purchased, Profile, Labels, All
  -da, --download-area DOWNLOAD_HELPER
                                  Perform download in specified areas (comma or
                                  space separated). Options: HighLights,
                                  Archived, Messages, Timeline, Pinned, Stories,
                                  Purchased, Profile, Labels, All Has preference
                                  over --posts for download action
  -la, --like-area LIKE_HELPER    Perform like/unlike in selected areas (comma
                                  or space separated). Options: Archived,
                                  Timeline, Pinned, Labels, All Has preference
                                  over --posts for like action
  -ft, --filter TEXT              Filter posts by regex (case-sensitive if
                                  uppercase characters included)
  -nf, --neg-filter TEXT          Filter posts to exclude those matching regex
                                  (case-sensitive if uppercase characters
                                  included)
  -sp, --scrape-paid              Scrape entire paid page (can take a very long
                                  time)
  -xc, --max-count INTEGER        Maximum number of posts to download (oldest
                                  non-duped first)
  -it, --item-sort [random|text-asc|text-desc|date-asc|date-desc|filename-asc|filename-desc]
                                  Changes media processing order before action
                                  (default: date asc or random)
  -e, --force-all, --dupe, --dupe-all
                                  Download all files regardless of database
                                  presence
  -eq, --force-model-unique, --dupe-model-unique, --dupe-model, --force_model_unique
                                  Only download files with media ids not present
                                  for the current model in the database
  -fl, --force-like, --force-like-toggle, --no-cache-like
                                  force toggling of posts to like or unlike
                                  status regardless of cache
  -lb, --label LABEL_HELPER       Filter by label (use helpers.label_helper to
                                  process)
  -be, --before ARROW_HELPER      Process posts at or before the given date
                                  (MM/DD/YYYY) for likes, unlikes, and downloads
  -af, --after ARROW_HELPER       Process posts at or after the given date
                                  (MM/DD/YYYY) for likes, unlikes, and downloads
  -mm, --mass-only / -ms, --mass-skip
                                  Flag for enabling/disabling mass content or promos
                                  [select one --mass-only or --mass-skip]
  -ok, --only-timed / -sk, --skip-timed
                                  Flag for enabling/disabling promotional or temporary posts
                                  [select one --only-timed or --skip-timed]

Automation Options:
  Control automated actions (like/unlike/download) and background execution
  -d, --daemon FLOAT              Run script in the background. Set value to
                                  minimum minutes between script runs. Overdue
                                  runs will run as soon as previous run finishes
  -a, --action ACTION_HELPER      Select batch action(s) to perform
                                  [like,unlike,download]. Accepts space or
                                  comma-separated list. Like and unlike cannot
                                  be combined.

User Selection Options:
  Specify users for scraping  with usernames, userlists, or blacklists
  -u, --usernames, --username USERNAME_HELPER
                                  Select which username to process (name,name2).
                                  Set to ALL for all users
  -eu, --excluded-username USERNAME_HELPER
                                  Select which usernames to exclude
                                  (name,name2). Has preference over --username
  -ul, --user-list, --userlist, --userlists, --user-lists TEXT
                                  Filter by userlist. Note: the lists
                                  'ofscraper.main', 'ofscraper.expired', and
                                  'ofscraper.active' are reserved and should not
                                  be the name of any list you have on OF
  -bl, --black-lists, --blacklist, --blacklists, --black-list TEXT
                                  Remove all users from selected list. Note: the
                                  lists 'ofscraper.main', 'ofscraper.expired',
                                  and 'ofscraper.active' are reserved and should
                                  not be the name of any list you have on OF

User List Filter Options:
  Filter users with options like price (current/renewal/regular/promo), free
  trial, promo availability, alongside userlist filters (include/exclude)
  -cp, --current-price [paid|free]
                                  Filter accounts based on either the
                                  subscription price, lowest claimable
                                  promotional price, or regular price
  -rp, --renewal-price [paid|free]
                                  Filter accounts based on either the lowest
                                  claimable promotional price, or regular price
  -gp, --regular-price [paid|free]
                                  Filter accounts based on the regular price
  -pp, --promo-price [paid|free]  Filter accounts based on either the lowest
                                  promotional price regardless of claimability,
                                  or regular price
  -lo, --last-seen-only / -ls, --last-seen-skip
                                  Flag for filtering accounts based on last seen being visible
                                  [select one --last-seen-only or --last-seen-skip]
  -fo, --free-trial-only / -fs, --free-trial-skip
                                  Flag for enabling/disabling accounts with free trial
                                  [must be normally paid]
                                  [select one --free-trial-only or --free-trial-skip]
  -po, --promo-only / -ps, --promo-skip
                                  Flag for enabling/disabling accounts with a claimable promo price
                                  [select one --promo-only or --promo-skip]
  -ao, --all-promo-only / --all-promo-skip
                                  Flag for enabling/disabling  accounts with any promo price
                                  [select one all-promo-only or --all-promo-skip]
  -ts, --active-subscription / -es, --expired-subscription
                                  Flag for enabling/disabling  accounts with active or expired subscription
                                  [select one --active-subscription or --expired-subscription]
  -ro, --renew-on / -rf, --renew-off
                                  Flag for enabling/disabling accounts set to renew renew flag on
                                  [select one --renew-on or --renew-off]

Advanced User List Filter Options:
  Precise user filtering with price ranges (current/renewal/regular/promo), last
  seen dates, expiration dates, and subscription dates
  -ppn, --promo-price-min INTEGER
                                  Filter accounts where the lowest promo price
                                  matches or falls above the provided value
  -ppm, --promo-price-max INTEGER
                                  Filter accounts where the lowest promo price
                                  matches or falls below the provided value
  -gpn, --regular-price-min INTEGER
                                  Filter accounts where the regular price
                                  matches or falls above the provided value
  -gpm, --regular-price-max INTEGER
                                  Filter accounts where the regular price
                                  matches or falls below the provided value
  -cpn, --current-price-min INTEGER
                                  Filter accounts where the current regular
                                  price matches or falls above the provided
                                  value
  -cpm, --current-price-max INTEGER
                                  Filter accounts where the current price
                                  matches or falls below the provided value
  -rpn, --renewal-price-min INTEGER
                                  Filter accounts where the renewal regular
                                  price matches or falls above the provided
                                  value
  -rpm, --renewal-price-max INTEGER
                                  Filter accounts where the renewal price
                                  matches or falls below the provided value
  -lsb, --last-seen-before TEXT   Filter accounts by last seen being at or
                                  before the given date (YYYY-MM-DD format)
  -lsa, --last-seen-after TEXT    Filter accounts by last seen being at or after
                                  the given date (YYYY-MM-DD format)
  -ea, --expired-after TEXT       Filter accounts by expiration/renewal being at
                                  or after the given date (YYYY-MM-DD format)
  -eb, --expired-before TEXT      Filter accounts by expiration/renewal being at
                                  or before the given date (YYYY-MM-DD format)
  -sa, --subscribed-after TEXT    Filter accounts by subscription date being
                                  after the given date (YYYY-MM-DD format)
  -sb, --subscribed-before TEXT   Filter accounts by sub date being at or before
                                  the given date (YYYY-MM-DD format)

Model Sort & Processing Order Options:
  Define the order in which models are displayed and processed for actions like
  liking posts, downloading content, or data gathering
  -st, --sort [name|subscribed|expired|current-price|renewal-price|regular-price|promo-price|last-seen]
                                  What to sort the model list by
  -ds, --desc                     Sort the model list in descending order

Advanced Search & Processing Options:
  Choose how usernames are searched, and define the order in which users are
  processed for actions
  -uf, --users-first, --user-first
                                  Process all users first rather than one at a
                                  time (affects --action)
  -fi, --individual               Search each username as a separate request
                                  when --username is provided
  -fl, --list                     Search entire enabled lists before filtering
                                  for usernames when --username is provided

Advanced Program Options:
  Advanced control of program behavior
  -nc, --no-cache                 Disable cache and forces consecutive api scan
  -nca, --no-api-cache            Forces consecutive api scan
  -k, --key-mode [cdrm|cdrm2|manual|keydb]
                                  Key mode override
  -dr, --dynamic-rules, --dynamic-rule [deviint|digitalcriminals|manual|generic|datawhores|riley|xagler|rafa]
                                  Dynamic signing
  -up, --update-profile           Get up-to-date profile info instead of using
                                  cache
  -ds, --download-script TEXT     runs a script post model download
                                  addional args sent to script username, model_id, media json ,and post json

Other options:
  -h, --help                      Show this message and exit.

Commands:
  manual       Manually download media by providing a list of urls or IDs
  metadata     Modify metadata or db files
  msg_check    Produces a media table from posts with filterable entries and quick downloads
  paid_check   Produces a media table from purchases with filterable entries and quick downloads
  post_check   Produces a media table from posts with filterable entries and quick downloads
  story_check  Produces a media table from stories and highlights with filterable entries and quick downloads
datawhores commented 3 months ago

Yeah actual show me the stack trace And Do ofscraper -v

datawhores commented 3 months ago

It doesn't really matter though You don't have the latest arguments added yesterday anyways --media-id should be in media content And --post-id

Maybe try uninstall and reinstall

datawhores commented 3 months ago

Do ofscraper -v before uninstalling and see if it changes on reinstall

JSylvia007 commented 3 months ago

Uninstall / Reinstall worked! Freaking Python...

Downloads are working and are fast as expected.

maxcom99 commented 3 months ago

3.10b is awesome btw, finally resolved my UI issues. thank you!!!