Sibusten / derpibooru-downloader

A downloader for imageboards running Philomena, such as Derpibooru.
MIT License
62 stars 6 forks source link

Throws exception on relative URIs in JSON API responses #55

Open iceman-p opened 2 years ago

iceman-p commented 2 years ago

ponerpics.org's json responses use relative URIs instead of URIs with a FQDN, which breaks the downloader. Their view_url's are things like "/img/view/2012/1/2/1_safe[...]" instead of "https://derpicdn.net/img/view/2012/1/2/1__[...]".

Sample output of https://ponerpics.org/api/v1/json/images/1 for illustration:

{"image":{"legacy_faves":1377,"derpi_faves":1377,"tag_count":37,"deletion_reason":null,"combined_score":1996,"aspect_ratio":1.0,"legacy_score":1996,"width":900,"orig_sha512_hash":null,"description":"","representations":{"full":"/img/view/2012/1/2/1.png","large":"/img/2012/1/2/1/large.png","medium":"/img/2012/1/2/1/medium.png","small":"/img/2012/1/2/1/small.png","tall":"/img/2012/1/2/1/tall.png","thumb":"/img/2012/1/2/1/thumb.png","thumb_small":"/img/2012/1/2/1/thumb_small.png","thumb_tiny":"/img/2012/1/2/1/thumb_tiny.png"},"size":812228,"mime_type":"image/png","animated":false,"derpi_score":1996,"score":12,"name":"1__safe_fluttershy_solo_cloud_happy_flying_upvotes+galore_artist-colon-speccysy_get_sunshine","combined_faves":1377,"height":900,"sha512_hash":"e31a01e5df99a0d7a0f036f2ffb3c7e1abda9996cb00938e0c9978069073c2af1928b2be79728ffa1221f14143166ac97df22857447afd32e3b7146d3b82f66e","tag_ids":[1458,15442,23275,23294,24249,24672,26776,27141,27724,27764,29630,33258,33983,34506,34678,36872,37319,38185,40482,41700,41916,42350,43338,43567,43587,46526,47596,80809,83982,93524,182100,187857,227349,321220,364605,449198,999999],"format":"png","hidden_from_users":false,"combined_downvotes":17,"wilson_score":0.6439530063653618,"legacy_downvotes":17,"updated_at":"2020-07-29T19:10:16","created_at":"2012-01-02T03:12:33","downvotes":0,"combined_upvotes":2013,"view_url":"/img/view/2012/1/2/1__safe_fluttershy_solo_female_pony_mare_pegasus_smiling_cute_wings_eyes+closed_spread+wings_flying_happy_cloud_signature_dead+source_sky_shyabetes_o.png","source_url":"https://speccysy.deviantart.com/art/Afternoon-Flight-215193985","uploader_id":2,"thumbnails_generated":true,"derpi_downvotes":17,"first_seen_at":"2012-01-02T03:12:33","duplicate_of":null,"duration":null,"legacy_upvotes":2013,"faves":7,"processed":true,"id":1,"comment_count":1,"intensities":{"ne":71.08886,"nw":79.232849,"se":70.149523,"sw":72.540942},"upvotes":12,"spoilered":false,"tags":["artifact","artist:speccysy","cloud","cloudy","cute","dead source","eyes closed","female","fluttershy","flying","happy","long hair","mare","messy mane","milestone","outdoors","pegasus","pony","safe","signature","sky","solo","stretching","sunlight","sunshine","upside down","wings","shyabetes","sweet dreams fuel","weapons-grade cute","smiling","spread wings","index get","derpibooru legacy","first fluttershy picture on derpibooru","one of the first","imported from derpibooru"],"uploader":"Derpi Imported","derpi_upvotes":2013},"interactions":[]}

When this happens, the following stack trace is printed before exit:

Unhandled exception: Dasync.Collections.ParallelForEachException: One or more errors occurred. (Invalid URI: The format of the URI could not be determined.)
 ---> System.UriFormatException: Invalid URI: The format of the URI could not be determined.
   at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
   at System.Uri..ctor(String uriString)
   at Sibusten.Philomena.Client.Images.PhilomenaImage.get_Name()
   at Sibusten.Philomena.Downloader.ImageDownloader.GetFileForPathFormat(IPhilomenaImage image, String filePath, Boolean isSvgImage)
   at Sibusten.Philomena.Downloader.ImageDownloader.GetFileForImage(IPhilomenaImage image)
   at Sibusten.Philomena.Client.Images.Downloaders.PhilomenaImageFileDownloader.Download(IPhilomenaImage downloadItem, CancellationToken cancellationToken, IProgress`1 progress)
   at Sibusten.Philomena.Client.Images.Downloaders.SequentialPhilomenaImageDownloader.Download(IPhilomenaImage image, CancellationToken cancellationToken, IProgress`1 progress)
   at Sibusten.Philomena.Client.Images.Downloaders.ConditionalImageDownloader.Download(IPhilomenaImage downloadItem, CancellationToken cancellationToken, IProgress`1 progress)
   at Sibusten.Philomena.Client.Images.Downloaders.SequentialPhilomenaImageDownloader.Download(IPhilomenaImage image, CancellationToken cancellationToken, IProgress`1 progress)
   at Sibusten.Philomena.Client.Images.Downloaders.ConditionalImageDownloader.Download(IPhilomenaImage downloadItem, CancellationToken cancellationToken, IProgress`1 progress)
   at Sibusten.Philomena.Client.Images.Downloaders.SequentialPhilomenaImageDownloader.Download(IPhilomenaImage image, CancellationToken cancellationToken, IProgress`1 progress)
   at Sibusten.Philomena.Client.Images.Downloaders.ParallelPhilomenaImageSearchDownloader.<>c__DisplayClass5_0.<<BeginDownload>b__1>d.MoveNext()
   --- End of inner exception stack trace ---
   at Sibusten.Philomena.Client.Images.Downloaders.ParallelPhilomenaImageSearchDownloader.BeginDownload(CancellationToken cancellationToken, IProgress`1 searchProgress, IProgress`1 searchDownloadProgress, IReadOnlyCollection`1 individualDownloadProgresses)
   at Sibusten.Philomena.Downloader.ImageDownloader.StartDownload(CancellationToken cancellation, IImageDownloadReporter downloadReporter)
   at Sibusten.Philomena.Downloader.Cmd.Commands.DownloadCommand.DownloadCommandFunc(DownloadArgs args)
   at System.CommandLine.Invocation.CommandHandler.GetResultCodeAsync(Object value, InvocationContext context)
   at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseParseErrorReporting>b__21_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass25_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass23_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__22_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseParseDirective>b__20_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseDebugDirective>b__11_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__10_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<<UseExceptionHandler>b__0>d.MoveNext()