Scrapper for DMM Seems to be Broken Again using the Latest Code

rickytor81 commented 4 years ago

The scrapper for DMM appears to be broken again. I have been using the latest code uploaded by @Wizell, it works well in the past few days. It seems the DMM scrapper broken again today, it is not returning anything even with the manual url.

zuko7177 commented 4 years ago

DMM scraper relies on grammerchecker.net to do translation. That site is not reliable. If that site is not working, DMM scraper will not work properly. See https://github.com/DoctorD1501/JAVMovieScraper/issues/325#issuecomment-674108330

rickytor81 commented 4 years ago

DMM scraper relies on grammerchecker.net to do translation. That site is not reliable. If that site is not working, DMM scraper will not work properly. See #325 (comment)

Thank you @zuko7177 for the reply. I am not sure if the issue relates to the translation. I have the option enable to force Japanese scrapping through "Scrape JAV Movies in Japanese Instead Of English". It works fine until yesterday. I tried also using manual url, but still returning nothing. Thanks for looking into it.

zuko7177 commented 4 years ago

I have a pull request to improve DMM scraping. https://github.com/DoctorD1501/JAVMovieScraper/pull/332 In the meantime, if you're familiar with the process you can clone my repo and try it out.

Also, take a look at https://github.com/jvlflame/Javinizer I found out about it recently. It's great.

rickytor81 commented 4 years ago

I have a pull request to improve DMM scraping. #332 In the meantime, if you're familiar with the process you can clone my repo and try it out.

Also, take a look at https://github.com/jvlflame/Javinizer I found out about it recently. It's great.

Thank you @zuko7177 ! With your latest repo, it works! The speed is also improved a lot!! I mean A LOT!!!

koonfoon commented 3 years ago

Hi @zuko7177 recently I tried to scrape from DMM using command line, but it gave me error:

Filename =  ../plex/ssni-852/ssni-852.mp4
Parsing with parsing profile = class moviescraper.doctord.controller.siteparsingprofile.specific.DmmParsingProfile
DMM Scraper: Search string --> https://www.dmm.co.jp/search/=/searchstr=ssni-852/
Scraping this webpage for movie: https://www.dmm.co.jp/mono/dvd/-/detail/=/cid=ssni852/?i3_ref=search&i3_ord=2
DMM Scraper: getting JP version at https://www.dmm.co.jp/mono/dvd/-/detail/=/cid=ssni852/?i3_ref=search&i3_ord=2
DMM Scraper: Title --> 華奢な少女の人生初！絶頂ポルチオ開発 巨根×膣中イキオーガズム 槙いずな
DMM Scraper: Plot  --> 槙いずな、人生初のポルチオ開発宣言。奥のさらに奥…ポルチオ徹底開発！！「奥ダメぇぇ！！！子宮が下がってるぅぅぅ…」ズボボッ！！極太バイブ、汗みどろ巨大ペニス喉マンコ拡張イラマ、前代未聞の超ケイレン絶頂3P！おま●こヒクヒク！巨根で抉じ開ける…快感電流ビッキーン！！腹筋ガクガク大痙攣。込み上がる快感オーガズム！極細クビレBODYがイキ乱れ、華奢な 少女が失神するまでケダモノ絶頂！
DMM Scraper: getting actresses from https://actress.dmm.co.jp/-/detail/=/actress_id=1059504/
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "org.jsoup.nodes.Element.attr(String)" because "actressThumbnailElement" is null
        at moviescraper.doctord.controller.siteparsingprofile.specific.DmmParsingProfile.scrapeActors(DmmParsingProfile.java:583)
        at moviescraper.doctord.model.Movie.<init>(Movie.java:139)
        at moviescraper.doctord.model.Movie.scrapeMovie(Movie.java:821)
        at moviescraper.doctord.Main.runScrape(Main.java:215)
        at moviescraper.doctord.Main.main(Main.java:114)

It seem like actress thumbnail is missing? DMM made changes again on their site source code? Please advise. Thank you.

rickytor81 commented 3 years ago

koonfoon, a quick containment is to disable actress scraper, "Scrapper's Settings/DMM/Scrape Actress". It works at least, then you need to manfully add actress info.

Of course, it is only a containment. Let's wait for comments from @zuko7177.

koonfoon commented 3 years ago

I am able to fix "actressThumbnailElement" is null error. The line of code that causing error:

// Error: unable to select the element
Element actressThumbnailElement = actressPage.select("tr.area-av30.top td img").first();

Changed the css query:

Element actressThumbnailElement = actressPage.select("span.p-section-profile__image img").first();

This is working on my dev/test environment. But I had accidentally update my actual environment that run this scraper to `JDK 16. 😭 Now I got this error:

Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.Comparator java.util.TreeMap.comparator accessible: module java.base does not "opens java.util" to unnamed module @5fb2de77

DoctorD1501 / JAVMovieScraper

Scrapper for DMM Seems to be Broken Again using the Latest Code #333