Closed dykn closed 4 years ago
Hi @dykn, I'll take a look asap!
@dykn I think that its not related to the tool itself but at some point the magadex server starts to throttle the requests.
@bake what do you think about it?
Is it possible it's an IP address block/ban? The VPS it's on doesn't have an option for a seconday/dynamic IP address, so I don't really have a way of knowing for sure. The error has persisted for over a week now, so I'm just making that assumption at this point.
I guess so, the error is due to a
json.Decode(response.Body)
where Body is not a json.
I'll check what is inside Body when this happen but I suppose that there's html inside.
Thank you for the mention! I think you're right, @Girbons, I'll add an additional error check tomorrow.
Thank you, it could be helpful.
Please try the latest commit. It only decodes if the status is OK and shows the message otherwise. While testing, I got 503. This reports a HTML page with the message "Please stop spamming the server", however the client replies with the default message of "Service Unavailable".
Thanks @bake, I'm going to try it out.
Newest error, using latest release:
./comics-downloader -url="https://mangadex.org/title/45112/tensei-kenja-wa-musume-to-kurasu" -format=cbz INFO[0000] Downloading... url="https://mangadex.org/title/45112/tensei-kenja-wa-musume-to-kurasu" ERRO[0000] could not unmarshal manga 45112: json: invalid number literal, trying to unmarshal """" into Number
I've just tested it and got the same error. However, using the client directly works:
func main() {
md := mangadex.New()
m, _, err := md.Manga("45112")
if err != nil {
log.Fatal(err)
}
fmt.Println(m.Title)
// Tensei Kenja wa Musume to Kurasu
}
I'll take a look tomorrow on this, also noted that is the same error that I get while running tests on mangadex.
FWIW it doesn't work for me with v0.21.2 but it does when I go run
(or make
) the program directly.
Same error while running the client directly
Might have something to do with MD using Cloudflare again. Could you please try to save the response? With something like httpcache:
c := httpcache.New(diskcache.New("cache", diskcache.NoExpiration))
md := mangadex.New(mangadex.WithHTTPClient(c.Client()))
I also noticed the output before the error message. Are you using the most recent commit?
The response seems to be ok:
HTTP/2.0 200 OK
Content-Security-Policy: upgrade-insecure-requests;
Content-Type: application/json
Date: Sun, 08 Mar 2020 13:24:21 GMT
Server: ngjit
Set-Cookie: __ddg1=7rR09689CrnSVVdbpEae; Domain=.mangadex.org; HttpOnly; Path=/; Expires=Mon, 08-Mar-2021 13:24:20 GMT
Vary: Accept-Encoding
{"manga":{"cover_url":"\/images\/manga\/45112.jpg?1580472947","description":"Meet Kento Oonari, salaryman in hell.\r\nOverworked by his bosses, his head surely whirled.\r\nHe dies, but meets up with a god called Raidel\r\nWho tells him that he’ll be sent to a new world.\r\n\r\nKento's fairly cautious - he’s read this before. \r\nAnd so, to the god, he begs for one thing:\r\n“I just want to live a slow life, nothing more.\r\n“Just give me a house and a skill for farming.”\r\n\r\nSo Kento is sent to a medieval age\r\nTo live a nice, slow life. But what a surprise!\r\nThe god gave him the powers of a Great Sage!\r\nAnd Kento used magic in front of commoner eyes!\r\n\r\nSo join us today, as we read of this man:\r\nHis cheat skills, his slow life, and his little girl clan.\r\n\r\n[url=https:\/\/ncode.syosetu.com\/n4302es\/]Original WN[\/url]","title":"Tensei Kenja wa Musume to Kurasu","artist":"Uso Uzaki\u3000","author":"Ryo Kotohira","status":1,"genres":[5,31,41,42,65],"last_chapter":"0","lang_name":"Japanese","lang_flag":"jp","hentai":0,"links":{"ap":"tensei-kenja-wa-musume-to-kurasu","mu":"162804","raw":"https:\/\/seiga.nicovideo.jp\/comic\/45295"}},"chapter":{"816959":{"volume":"","chapter":"2","title":"","lang_code":"fr","group_id":11025,"group_name":"Shiro FanTrad","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582915077},"812849":{"volume":"","chapter":"2","title":"","lang_code":"ru","group_id":11141,"group_name":"S1nGeR","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582389516},"816906":{"volume":"","chapter":"1.3","title":"","lang_code":"fr","group_id":11025,"group_name":"Shiro FanTrad","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582906719},"811874":{"volume":"","chapter":"1.3","title":"","lang_code":"ru","group_id":11141,"group_name":"S1nGeR","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582284641},"815967":{"volume":"","chapter":"1.2","title":"","lang_code":"fr","group_id":11025,"group_name":"Shiro FanTrad","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582755619},"811873":{"volume":"","chapter":"1.2","title":"","lang_code":"ru","group_id":11141,"group_name":"S1nGeR","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582284592},"810596":{"volume":"","chapter":"1.1","title":"","lang_code":"fr","group_id":11025,"group_name":"Shiro FanTrad","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582159890},"811872":{"volume":"","chapter":"1.1","title":"","lang_code":"ru","group_id":11141,"group_name":"S1nGeR","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582284535},"810594":{"volume":"","chapter":"1","title":"","lang_code":"fr","group_id":11025,"group_name":"Shiro FanTrad","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582159819},"811870":{"volume":"","chapter":"1","title":"","lang_code":"ru","group_id":11141,"group_name":"S1nGeR","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582284330},"822479":{"volume":"1","chapter":"2.1","title":"(part two)","lang_code":"gb","group_id":861,"group_name":"hidamari","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1583590563},"812800":{"volume":"1","chapter":"2","title":"(part one)","lang_code":"gb","group_id":861,"group_name":"hidamari","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582383029},"814802":{"volume":"1","chapter":"2","title":"Part 1","lang_code":"id","group_id":6390,"group_name":"Aoirosubs","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582614362},"805995":{"volume":"1","chapter":"1.3","title":"(part four)","lang_code":"gb","group_id":861,"group_name":"hidamari","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1581768408},"814657":{"volume":"1","chapter":"1.3","title":"Part 4","lang_code":"id","group_id":6390,"group_name":"Aoirosubs","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582597982},"805965":{"volume":"1","chapter":"1.2","title":"(part three)","lang_code":"gb","group_id":861,"group_name":"hidamari","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1581767999},"814655":{"volume":"1","chapter":"1.2","title":"Part 3","lang_code":"id","group_id":6390,"group_name":"Aoirosubs","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582597867},"805092":{"volume":"1","chapter":"1.1","title":"(Part two)","lang_code":"gb","group_id":861,"group_name":"hidamari","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1581682832},"809097":{"volume":"1","chapter":"1.1","title":"Part 2","lang_code":"id","group_id":6390,"group_name":"Aoirosubs","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1582024322},"803611":{"volume":"1","chapter":"1","title":"(Part one)","lang_code":"gb","group_id":861,"group_name":"hidamari","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1581467928},"808743":{"volume":"1","chapter":"1","title":"Part 1","lang_code":"id","group_id":6390,"group_name":"Aoirosubs","group_id_2":0,"group_name_2":null,"group_id_3":0,"group_name_3":null,"timestamp":1581980418}},"status":"OK"}```
There seems to have been a breaking change in Go 1.14 which disallows invalid (in this case missing, see volume
in chapter
) numbers in json.Number
. I currently don't have the time to decide and upgrade to int64
, string
s or a custom type. This also requires breaking changes in the library. There will be an update during the next week and for now you could use 1.13 to build the program.
Please try the new commit, I've added a wrapper.
Going to try the new commit thanks!
Hey @bake thanks a lot, It works, I'm going to release a new version
v0.21.3 released
INFO[0000] Downloading... url="https://mangadex.org/title/907/i-am-a-hero" ERRO[0530] could not get chapter 511880: could not decode response: invalid character '<' looking for beginning of value
Tried multiple different mangas/URLs.
Thanks.