search

xbapps / xbvr

Tool to organize and stream your VR porn library
336 stars 130 forks source link

panic due to unhanded condition in scrape.VirtualRealPornSite #1001

Open wheresmytissue opened 1 year ago

wheresmytissue commented 1 year ago

Problem

Program crashed when scraping.

Analysis

I think this is due to a simple check on the body return values per this stack overflow post

From my understanding of the log output below, it is occurring here I think if you check the jsonResult var

Potential fix

perhaps something like this

if jsonResult != nil {
  // continue parsing..
} else {
  // elegantly exit function
}

Raw Logs Below

time="2022-10-28T02:19:56Z" level=info msg="visiting https://vrphub.com/oil-worship-ass/"
panic: interface conversion: interface {} is nil, not string

goroutine 1241669 [running]:
github.com/xbapps/xbvr/pkg/scrape.VirtualRealPornSite.func1.6(0xc00bef9340?, 0xc014942480)
        /go/src/pkg/scrape/virtualrealporn.go:78 +0x485
github.com/gocolly/colly.(*HTMLElement).ForEach.func1(0xc0031663d8?, 0xc01e1a9110)
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/htmlelement.go:98 +0x83
github.com/PuerkitoBio/goquery.(*Selection).Each(0xc01e1a90e0, 0xc008d10ec0)
        /go/pkg/mod/github.com/!puerkito!bio/goquery@v1.8.0/iteration.go:10 +0x46
github.com/gocolly/colly.(*HTMLElement).ForEach(0xc0149422a0, {0x140cc1f?, 0x13af7dc?}, 0xc008d10fc8)
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/htmlelement.go:96 +0x78
github.com/xbapps/xbvr/pkg/scrape.VirtualRealPornSite.func1(0xc0149422a0)
        /go/src/pkg/scrape/virtualrealporn.go:74 +0x42f
github.com/gocolly/colly.(*Collector).handleOnHTML.func1(0xc003166388?, 0xc01e1a8d80)
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:963 +0x78
github.com/PuerkitoBio/goquery.(*Selection).Each(0xc01e1a8d50, 0xc008d11280)
        /go/pkg/mod/github.com/!puerkito!bio/goquery@v1.8.0/iteration.go:10 +0x46
github.com/gocolly/colly.(*Collector).handleOnHTML(0xc007f56820, 0xc00bef9340)
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:953 +0x24f
github.com/gocolly/colly.(*Collector).fetch(0xc007f56820, {0xa?, 0x3?}, {0x13b0737, 0x3}, 0x1, {0x0?, 0x0}, 0x41607f?, 0xc003a89860, ...)
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:623 +0x6ad
github.com/gocolly/colly.(*Collector).scrape(0xc007f56820, {0xc00cdc7090, 0x43}, {0x13b0737, 0x3}, 0x1, {0x0, 0x0}, 0x0, 0x0, ...)
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:535 +0x6b1
github.com/gocolly/colly.(*Collector).Visit(0xc00cdc7090?, {0xc00cdc7090?, 0x13af7dc?})
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:412 +0xa6
github.com/xbapps/xbvr/pkg/scrape.VirtualRealPornSite.func4(0xc01bbde780?)
        /go/src/pkg/scrape/virtualrealporn.go:211 +0xe5
github.com/gocolly/colly.(*Collector).handleOnHTML.func1(0xc019fb7630?, 0xc003a89800)
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:963 +0x78
github.com/PuerkitoBio/goquery.(*Selection).Each(0xc003a897d0, 0xc008d11758)
        /go/pkg/mod/github.com/!puerkito!bio/goquery@v1.8.0/iteration.go:10 +0x46
github.com/gocolly/colly.(*Collector).handleOnHTML(0xc007f569c0, 0xc0184c5a80)
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:953 +0x24f
github.com/gocolly/colly.(*Collector).fetch(0xc007f569c0, {0xa?, 0x3?}, {0x13b0737, 0x3}, 0x1, {0x0?, 0x0}, 0xc01fcf1f00?, 0xc00839c780, ...)
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:623 +0x6ad
github.com/gocolly/colly.(*Collector).scrape(0xc007f569c0, {0xc018657fc0, 0x39}, {0x13b0737, 0x3}, 0x1, {0x0, 0x0}, 0x0, 0x0, ...)
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:535 +0x6b1
github.com/gocolly/colly.(*Collector).Visit(0xc00595b200?, {0xc018657fc0?, 0x58?})
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:412 +0xa6
github.com/xbapps/xbvr/pkg/scrape.VirtualRealPornSite.func3(0xc015c92e40?)
        /go/src/pkg/scrape/virtualrealporn.go:203 +0x75
github.com/gocolly/colly.(*Collector).handleOnHTML.func1(0xc00069a678?, 0xc00839c690)
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:963 +0x78
github.com/PuerkitoBio/goquery.(*Selection).Each(0xc00329b9b0, 0xc008d11bf0)
        /go/pkg/mod/github.com/!puerkito!bio/goquery@v1.8.0/iteration.go:10 +0x46
github.com/gocolly/colly.(*Collector).handleOnHTML(0xc007f569c0, 0xc0008174c0)
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:953 +0x24f
github.com/gocolly/colly.(*Collector).fetch(0xc007f569c0, {0xa?, 0x3?}, {0x13b0737, 0x3}, 0x1, {0x0?, 0x0}, 0x29?, 0xc007f55500, ...)
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:623 +0x6ad
github.com/gocolly/colly.(*Collector).scrape(0xc007f569c0, {0xc004368690, 0x29}, {0x13b0737, 0x3}, 0x1, {0x0, 0x0}, 0x0, 0x0, ...)
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:535 +0x6b1
github.com/gocolly/colly.(*Collector).Visit(0xc007f569c0?, {0xc004368690?, 0x19?})
        /go/pkg/mod/github.com/gocolly/colly@v1.2.0/colly.go:412 +0xa6
github.com/xbapps/xbvr/pkg/scrape.VirtualRealPornSite(0x0?, 0x1, {0xc0088ec000, 0x53ce, 0x5c00}, 0xc0086d41e0, {0x13d2ed0, 0xf}, {0x13d2138, 0
xf}, ...)
        /go/src/pkg/scrape/virtualrealporn.go:218 +0x48a
github.com/xbapps/xbvr/pkg/scrape.VirtualRealPorn(0x0?, 0x0?, {0xc0088ec000?, 0x0?, 0x0?}, 0x0?)
        /go/src/pkg/scrape/virtualrealporn.go:231 +0x57
created by github.com/xbapps/xbvr/pkg/tasks.runScrapers
        /go/src/pkg/tasks/content.go:109 +0x49b
toshski commented 1 year ago

If your log is correct, I don't understand how you are scraping a vrphub.com scene with the virtualrealporn scraper, the virtualrealporn scraper does not scrape that site. I'm not surprised it didn't work as vrphub pages will be completely different and if the scraper didn't crash there it probably would somewhere else.

Do you have any mods or redirects. Are you sure the log message with the vrphub.com scene is the one that caused the crash. Maybe try scraping just the one single site (option on the 3 dots) rather than all of them to confirm the site with a problem. I'm not aware of anyone else having issues with this.

theRealKLH commented 1 year ago

@wheresmytissue cleaning up "dead" issues. VirtualRealPorn scraper has been updated (not for this issue) if you still have the problem, reopen with more information.

theRealKLH commented 1 year ago

whoops.. jumped the gun on this one.. (scraper was updated day before issue logged).

@wheresmytissue Please give more information (still an issue? xbvr version? answer @toshski, etc..) or I will close again in a day or two.