Closed bnkai closed 4 months ago
ThePornDB.yml: metadataapi.net api changed, "parse" no longer delivers tags. if i understand it right first parse to find the scene, then use the id / url to geht the rest of the scene data
The scraper was updated 4 days ago #509 for the tag issue, are you sure you have latest version?
sorry somehow i misse the "The" and only saw PornDB.ym - 16 days ago
Clips4Sale has changed the date format on their sites to DD/MM/YY and so the date parsing is now broken.
javdb is currently broken, confirmed by bnkai on discord and i'm just posting it here for posterity.
Thirdmovies.com does not seem to return any data, ztod contained within the same scraper, does still seem to work without issue.
RealityKings not working (with Brazzers.yml) , returns a error exec: "google-chrome": executable file not found in %PATH% VixenNetwork.yml also not working (don't know how to use the .py provided)
RealityKings not working (with Brazzers.yml) , returns a error exec: "google-chrome": executable file not found in %PATH% VixenNetwork.yml also not working (don't know how to use the .py provided)
Neither of the scrapers mentioned are broken. https://github.com/stashapp/CommunityScrapers#communityscrapers provides instructions for both cases (CDP setup and where to put the scraper files). For more info either have a look at the in-app help section of stash or visit the discord channel
MyDirtyHobby scraper not returning results. click the button to scrape next to url, spins, then nothing happens.
transangels and loveherfeet no longer scraping cover images
transangels and loveherfeet no longer scraping cover images
Can you provide a url where loveherfeet doesnt work? The image part seems fine from a few scenes i tried
transangels and loveherfeet no longer scraping cover images
Can you provide a url where loveherfeet doesnt work? The image part seems fine from a few scenes i tried
https://www.loveherfeet.com/tour/trailers/Did-you-post-my-feet-pictures-kayla-kayden-footjob.html https://www.loveherfeet.com/tour/trailers/Aunt-Lexis-feet-lexi-luna-footjob.html https://www.loveherfeet.com/tour/trailers/Suck-My-Naughty-Feet-Underneath-The-Tree-Violet-Starr-Feet.html These all missed the image when scraped
https://www.loveherfeet.com/tour/trailers/Did-you-post-my-feet-pictures-kayla-kayden-footjob.html https://www.loveherfeet.com/tour/trailers/Aunt-Lexis-feet-lexi-luna-footjob.html https://www.loveherfeet.com/tour/trailers/Suck-My-Naughty-Feet-Underneath-The-Tree-Violet-Starr-Feet.html These all missed the image when scraped
All of the above work fine for me. Can you double check that you have the latest version of the loveherfeet scraper (# Last Updated May 16, 2021) ?
All of the above work fine for me. Can you double check that you have the latest version of the loveherfeet scraper (# Last Updated May 16, 2021) ?
That was the problem, I missed the updated scraper. Sorry and thanks, bnkai
When loading stash with the latest scrapers I see these errors.
ERRO[2021-07-01 11:11:58] Error loading scraper <path>\HelixStudios.yml: invalid post-process action
ERRO[2021-07-01 11:11:58] Error loading scraper <path>\MaxHardcore.yml: invalid post-process action
Might not be the right thread but figured it was worth pointing out.
@nobofernandez some scrapers require development releases of Stash, so this is expected. v0.8.0 should be released soon, when you update it will clear those errors.
TeamskeetAPI.py - Seems to be an issue with some date conversion. I'm trying to see if I can teach myself python and figure it out, but I've always coded for fun so I don't know how far I'll get.
ERRO[2021-07-01 21:07:12] scraper: Asking the API...
ERRO[2021-07-01 21:07:12] scraper: Traceback (most recent call last):
ERRO[2021-07-01 21:07:12] scraper: File "C:\Users\<user>\.stash\scrapers\TeamskeetAPI.py", line 90, in <module>
ERRO[2021-07-01 21:07:12] scraper: date = datetime.strptime(scene_api_json.get(
ERRO[2021-07-01 21:07:12] scraper: File "C:\Users\<user>\AppData\Local\Programs\Python\Python39\lib\_strptime.py", line 568, in _strptime_datetime
ERRO[2021-07-01 21:07:12] scraper: tt, fraction, gmtoff_fraction = _strptime(data_string, format)
ERRO[2021-07-01 21:07:12] scraper: File "C:\Users\<user>\AppData\Local\Programs\Python\Python39\lib\_strptime.py", line 349, in _strptime
ERRO[2021-07-01 21:07:12] scraper: raise ValueError("time data %r does not match format %r" %
ERRO[2021-07-01 21:07:12] scraper: ValueError: time data '2017-08-25T18:00:00-04:00' does not match format '%Y-%m-%dT%H:%M:%S.%f%z'
ERRO[2021-07-01 21:07:12] could not unmarshal json: EOF
Fit18 is broken as reported in #700 A python scraper from #702 should fix the issue
nympho.yml is not working for scraping descriptions (at least on swallowed.com). I can't work out the x-path for it.
An example URL can be found here: https://tour.swallowed.com/view/56/cock-gargling-fun-with-morgan-karlee-and-aj
The text seems to be separated into 3 divs
JacquieEtMichelTV.py has some errors.
[Scrape / JacquieEtMicaelTV] IndexError: list index out of range
[Scrape / JacquieEtMicaelTV] details = tree.xpath("//div[@class='video-description']/p")[0]
[Scrape / JacquieEtMicaelTV] File "JacquieEtMichelTV.py", line 51, in <module>
[Scrape / JacquieEtMicaelTV] Traceback (most recent call last):
When I change line 51 from [0] to [1] it sometimes work. However sometimes it doesn't and in log can be 2 errors:
I reported this in the discord a few days ago, but a few of the sites the AdultEmpireCash scraper (MyPervyFamily, TouchMyWife, and FilthyKings at least) changed layouts and url structures so they no longer work.
Tushy Raw is broken. The page loads some js first now so the python scraper doesnt have access to the needed
In Nympho.yml I'm consistently having AllAnal or OnlyAnal work but TrueAnal 95% of the time when I enter the URL scrape button stays grayed out. I have entered a user agent in the scraper settings and it's working consistently on the other sites. Anyone else experiencing this behavior?
Example Url to scrap - https://tour.trueanal.com/scenes/baileys-stretchy-asshole
In Nympho.yml I'm consistently having AllAnal or OnlyAnal work but TrueAnal 95% of the time when I enter the URL scrape button stays grayed out. I have entered a user agent in the scraper settings and it's working consistently on the other sites. Anyone else experiencing this behavior?
Example Url to scrap - https://tour.trueanal.com/scenes/baileys-stretchy-asshole
The existing scraper is looking for URL's that start withtrueanal.com/view/
and the site mostly uses trueanal.com/scenes/
. I manually changed and tested and it was working. I tried to get the image scraper working as well but then it stopped working and now I just get
Processing scene:
in the trace log without any other related messages. I'm new to working with scrapers if anyone has any ideas. When I modified the URL path before everything else for Title, Performers, Summary, and Date pulled correctly except for the image.
Transangels - no longer working due to recent website update. Sits within RealityKingsOL.yml
Transangels - no longer working due to recent website update. Sits within RealityKingsOL.yml
For now, till we decide what to do you can edit the url and scrape with
the MindGeekAPI scraper (needs python + requests )
Edit: It seems we can move the site to MindGeek.yml now, so removing it from RealityKingsOL.yml and adding to MindGeek.yml should be enough.
@byertsy you can apply the PR or do the changes manually from here https://github.com/stashapp/CommunityScrapers/pull/807/files and then reload scrapers
Worked like a charm 👍
MyDirtyHobby scraper works in principle but does not retrieve the scene image. Instead, it shows an icon as if the scene image was not readable to Stash. If you click "Apply" an error message comes up and nothing is saved to the database. Current workaround to be able to use the scraper at all: Take over all data except the image, then you can "Apply" at least that part. Image remains empty though.
Edit: Test-URL: https://de.mydirtyhobby.com/profil/4529516-MilaElaine/videos/5654262-Mein-1-Outdoor-Fick-2019
Hi Guys any update for tushyraw scrapper?
Hi Guys any update for tushyraw scrapper?
I am having a look but i'll probably scrap it and make one from scratch. Till then i would suggest using ThePornDB scraper as it seems to have even recent scenes
Edit: PR available with a new vixenNetwork scraper #849
A stupid question maybe. How can once use ThePornDB scrapper? It looks like API needs authentication token.
A stupid question maybe. How can once use ThePornDB scrapper? It looks like API needs authentication token.
Create a account on metadataapi.net, after you can create a API key (in API Tokens)
That's clear. But how can I use this token directly in stash? Do I need to change something directly in yaml?
That's clear. But how can I use this token directly in stash? Do I need to change something directly in yaml?
Yes, open the yml with a notepad, go to driver section (bottom). Add your key after Bearer. It need to look like this:
driver:
headers:
- Key: User-Agent
Value: stashjson/1.0.0
- Key: Authorization # Uncomment and add a valid API Key after the `Bearer ` part
Value: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Thanks. I actually didn't see this even though I had it open.
The iWantClips scraper occasionally fails with the message "JSON.parse: unexpected character at line 1 column 1 of the JSON data". Nothing is produced in the logs while on "Info", and it only fails some of the time.
Edit: Seemed to be a problem with my CDP container. Increasing the SHM size to 2G solved this.
I'm trying to get 21sextreme to work but failing. EG: https://21sextreme.com/en/video/114967/Give-Me-Five
I'm trying to get 21sextreme to work but failing. EG:
https://21sextreme.com/en/video/114967/Give-Me-Five
Most of the Gamma sites seem to be moving to api based sites. Remove the 21sextreme url from the GammaEntertainment.yml and use the below Algolia_Sextreme.yml (you would need to have the Algolia.py file and the py_common folder from the repo)
name: "Sextreme"
sceneByURL:
- action: script
url:
- https://www.21sextreme.com/en/video
script:
- python
- Algolia.py
- 21sextreme
sceneByFragment:
action: script
script:
- python
- Algolia.py
- 21sextreme
sceneByName:
action: script
script:
- python
- Algolia.py
- 21sextreme
- searchName
sceneByQueryFragment:
action: script
script:
- python
- Algolia.py
- 21sextreme
- validName
# Last Updated January 23, 2022
the Colette scraper is broken and gives me an
Error Get "url": x509: Certificate has expired or is not yet valid:
message when i try to scrape a scene
the Colette scraper is broken and gives me an
Error Get "url": x509: Certificate has expired or is not yet valid:
message when i try to scrape a scene
they have issues with their ssl certificate
check for insecure...
option like below
Since a couple of days ago the kink.yml scraper for kink.com no longer works for me. Everytime i copy a link for scraping that scene i get: http error 403:Forbidden
Edit: Also many Sites seem to have changed their tags. They are now scraped with a comma behind each tag. Example: Sweetheartvideo.com
Since a couple of days ago the kink.yml scraper for kink.com no longer works for me. Everytime i copy a link for scraping that scene i get: http error 403:Forbidden
Edit: Also many Sites seem to have changed their tags. They are now scraped with a comma behind each tag. Example: Sweetheartvideo.com
Came here to say the same thing. 403:Forbidden
error.
Since a couple of days ago the kink.yml scraper for kink.com no longer works for me. Everytime i copy a link for scraping that scene i get: http error 403:Forbidden Edit: Also many Sites seem to have changed their tags. They are now scraped with a comma behind each tag. Example: Sweetheartvideo.com
Came here to say the same thing.
403:Forbidden
error.
For kink there is already a PR here https://github.com/stashapp/CommunityScrapers/pull/875 The scraper now requires CDP so make sure that is configured properly in stash
Since a couple of days ago the kink.yml scraper for kink.com no longer works for me. Everytime i copy a link for scraping that scene i get: http error 403:Forbidden
An updated Kink scraper was merged so that should be fixed now (requires CDP now)
Edit: Also many Sites seem to have changed their tags. They are now scraped with a comma behind each tag. Example: Sweetheartvideo.com
PR #881 should fix the issue If any of the affected Sites are not listed below (the PR covers the below) please make a new post and paste them
- babesnetwork.com/scene/
- devianthardcore.com/scene/
- doghousedigital.com/
- familyhookups.com/scene/
- girlgrind.com/scene
- kinkyspa.com/scene/
- lookathernow.com/scene/
- milehighmedia.com/scene/
- mofosnetwork.com/scene/
- realityjunkies.com/scene/
- rk.com/scene/
- squirted.com/scene/
- sweetheartvideo.com/scene/
- sweetsinner.com/scene/
- teenslovehugecocks.com/scene/
- twistys.com/scene/
- twistysnetwork.com/scene/
wankittnow Details: //meta[@property="og:description"]/@content Will only pull back partial info: e.g. https://www.wankitnow.com/videoentry/smoking-hot-milf-allows-step-son-to-watch-her-play-with-her-dildo-in-the-kitchen/
"What's this? Your step mum is in the kitchen and taking off her panties? Now she's placing herself ontop of the kitchen worktop and grabbing a glass dido, "
I've changed locally to: Details: //div[@class="post_excerpt"]//p and this pulls the full details.
I'm not overly familiar with scraping or the edit process here so am not comfortable with editing - hope this is of some help. Please let me know if I'm incorrect. Noob.
Thanks.
XBVR DB
Dates on most of the scenes I scrape from my database are off by one.
It seems that XBVR sqlite database contains the release_date
column in local timezone and the Python script parses it to UTC timezone. This bug is only visible if your local timezone offset is positive from UTC
Edit: Pull request https://github.com/stashapp/CommunityScrapers/pull/951
elegantangel.com | AdultEmpireCash.yml
It works, but it never sets the studio (Elegant Angel) when I use it to scrape a scene from a link. Scraping a movie sets the studio correctly.
For Tonight's Girlfriend, it doesn't seem to be grabbng the title. It just says Performers Name and then Tonights Girlfriend. It's also not grabbing the tags. I think this is new, the public facing site didn't show tags before.
AdultTime - Error: "could not unmarshal json from script output: EOF"
Plumperpass is putting the details data in the title field.
Menatplay doesn't work anymore
Any issues with scrapers not working should be mentioned here The name of the scraper, the xpath or part not working would be appretiated.
Known Issues
nhentai scraper is broken ( blocked/detected by site / CF ?)
updated 2022-09-25