SDWebImage from LPLinkMetadata in UICollectionViewCell

[inPhilly]

New Issue Checklist

• [X ] I have read and understood the CONTRIBUTING guide
• [X ] I have read the Documentation
• [X ] I have searched for a similar issue in the project and found none

Issue Info

Info	Value
Platform Name	ios
Platform Version	13.2.1
SDWebImage Version	5.0.0
Integration Method	carthage
Xcode Version	Xcode 11
Repro rate	n/a
Repro with our demo prj	n/a
Demo project link	n/a

Issue Description and Steps

This is more of a question than an issue. Can LPLinkMetaData be used with SDWebImage when loading data into a UICollectionViewCell, if one does not choose to use LPLinkView?

[dreampiggy]

I don't check what these class is. But actually, SDWebImage can load Any Resource with custom loader, render Any Target(View) with custom Image Class and Animated Player. No limit to UIView/NSURL concreate class. You can check our wiki about this design

Or, tomorrow I'll have a check for your described third party integration.

[inPhilly]

Here is a link to describe LPLinkMetaData - it is new in iOS 13 from Apple: https://developer.apple.com/documentation/linkpresentation/lplinkmetadata?language=objc

It uses NSItemProvider, so I need to load and cache image from NSItemProvider.

I am pretty new at this. If you have any suggestion of how I could load into a UICollectionViewCell UIImageView using SDWebImage, I would really appreciate it!

[dreampiggy]

See exist custom loader for Photos PHAsset: https://github.com/SDWebImage/SDWebImagePhotosPlugin

Exist custom View like FLAnimatedImageView: https://github.com/SDWebImage/SDWebImageFLPlugin

Since we use protocol based API design, it's always possible to build the block. See wiki page for detail usage, and with the documentation site about the API.

Or maybe I can just create one SDWebImageLinkPresentationPlugin (sounds verbose, SDWebImageLinkPlugin ?) demo show case if you can provide simple usage code.

[inPhilly]

//UICollectionViewCell Subclass

- (void)configureWithLinkMetadata:(LPLinkMetadata*)linkMetadata  API_AVAILABLE(ios(13.0)){

    NSString *linkURLString = linkMetadata.originalURL.absoluteString;

    NSString *linkTitle = linkMetadata.title;
    if (!(linkTitle.length > 0)) {
        linkTitle = linkURLString;
    }
    self.linkTitleLabel.text = linkTitle;

    [self.linkImageView sd_setImageWithNSItemProvider:linkMetadata.imageProvider
    placeholderImage:[UIImage imageNamed:@"transparentSquare"]
           completed:nil];
}

- (void)prepareForReuse {

    [super prepareForReuse];
    self.linkTitleLabel.text = nil;
    [self.linkImageView sd_cancelCurrentImageLoad];
    self.linkImageView.image = nil;
}

[dreampiggy]

I saw LPLinkView already have one initializer init(url:). Which just what you need ?

It seems already have one nice setImageWith(url:) method, isn't it ? Or it's just a placeholder which have limitation or traps ? Sorry I don't have a try for this yet.

[inPhilly]

I don't want to use LPLinkView. I want to get the UIImage from the LPLinkMetadata and use that.

[inPhilly]

LPLinkMetadata gives NSItemProvider. We can get UIImage asynchronously from that.

[dreampiggy]

From the API design, I think that

A URL -> Async Query with LPMetadataProvider -> Get LPMetadata -> Contains Image NSItemProvider -> So, what's in the item provider ? That it's a abstract object, it's that NSData already ? UIImage ? Anything reason you want to use with SDWebImage's components ?

SDWebImage have a success decoding system (NSData -> UIImage), URL Loading system (Network request), and Cache System (Memory/Disk Cache for Image and its Data). Tell me what this plugin support for LinkPresentation can be ?

[dreampiggy]

If the NSItemProvider's object is:

• Image HTTP URL, good, already supported via SDWebImageManager.
• Image NSData representation. good, already supported via UIImage+MultiFormat or decoder.
• Image with UIImage object, so it's end.
• Some strange internal Apple subclass, we can not do much for this, actually, until there are exist API to generate one Image Representation(can be any CGImage, UIImage, CVPixelBuffer, IOSurface, etc. Bitmap form of Image in logic idea)

[inPhilly]

I want to use SDWebImage because the image is loaded from NSItemProvider asyncronously. I want to be able to cancel the loading of the image in prepareForReuse to avoid cell reuse issues. I would also like to utilize the Cache System in the same way as for loading UIImage from NSURL.

[dreampiggy]

because the image is loaded from NSItemProvider asyncronously

So, my little question, How ? Or, any API ? This NSItemProvider is a abstract class, which use id<NSCoding>. How do I check or know it representation Image, but not text, video, audio, any other things ?

[inPhilly]

- (void)processLinkImageProvider:(NSItemProvider*)linkImageProvider withCompletion:(void (^) (UIImage *linkImage))completionBlock API_AVAILABLE(ios(13.0)) {

    if ([linkImageProvider hasItemConformingToTypeIdentifier:(NSString *)kUTTypeJPEG]) {
        [linkImageProvider loadDataRepresentationForTypeIdentifier:(NSString *)kUTTypeJPEG completionHandler:^(NSData * _Nullable linkImageData, NSError * _Nullable error) {

            [self processLinkImageData:linkImageData withCompletion:completionBlock];
        }];
    }
    else if ([linkImageProvider hasItemConformingToTypeIdentifier:(NSString *)kUTTypePNG]) {
        [linkImageProvider loadDataRepresentationForTypeIdentifier:(NSString *)kUTTypePNG completionHandler:^(NSData * _Nullable linkImageData, NSError * _Nullable error) {

            [self processLinkImageData:linkImageData withCompletion:completionBlock];
        }];
    }
    else if ([linkImageProvider hasItemConformingToTypeIdentifier:(NSString *)kUTTypeBMP]) {
        [linkImageProvider loadDataRepresentationForTypeIdentifier:(NSString *)kUTTypeBMP completionHandler:^(NSData * _Nullable linkImageData, NSError * _Nullable error) {

            [self processLinkImageData:linkImageData withCompletion:completionBlock];
        }];
    }
    else if ([linkImageProvider hasItemConformingToTypeIdentifier:(NSString *)kUTTypeGIF]) {
        [linkImageProvider loadDataRepresentationForTypeIdentifier:(NSString *)kUTTypeGIF completionHandler:^(NSData * _Nullable linkImageData, NSError * _Nullable error) {

            [self processLinkImageData:linkImageData withCompletion:completionBlock];
        }];
    }
    else if ([linkImageProvider hasItemConformingToTypeIdentifier:(NSString *)kUTTypeTIFF]) {
        [linkImageProvider loadDataRepresentationForTypeIdentifier:(NSString *)kUTTypeTIFF completionHandler:^(NSData * _Nullable linkImageData, NSError * _Nullable error) {

            [self processLinkImageData:linkImageData withCompletion:completionBlock];
        }];
    }
    else {
        [self processLinkImageData:nil withCompletion:completionBlock];
    }
}

[dreampiggy]

From your code, seems the NSItemProvider generated by LPMetadata, can only be one of compressed Data format, right ? Like JPEG/GIF/BMP/HEIF...

So, maybe we just need 2 method call, one is a huge switch case to check UTI type, another is call to generate NSData and pass to

[UIImage sd_imageWthData:data]

Sounds not hard. And you don' need any repeat code, just check if UTI in our coder supported array, they call same method is OK

[inPhilly]

Not sure if it could be other things, but those are the only ones I would be interested in. Ultimately, I want to be able to pass NSItemProvider and a placeholder image to set a cell's UIImageView, and also be able to cancel that load in prepareForReuse. I'm not sure how this can be done.

[dreampiggy]

I'm writing on iPad. Tomorrow I'll hav a detail demo app test for its behavior. Sounds really strange for this API design. If all the LPMetadata image item provider is a NSData, why not just use a NSURL here ? Apple's API design is strange.

[inPhilly]

NSURL isn't an image url. It's just a regular NSURL.

[dreampiggy]

But NSItemProvider can load Any id<NSCoding>, even a UIView, CALayer, right ? That API design is more generatic, there may be some reason (for example, that image may not even a URL), need to investigate and test on real device.

[inPhilly]

Apple does not want the image URL exposed to the public or stored. Link URLs can change often. This is for link previews. NSItemProvider from LPLinkMetadata should probably be an image.

[inPhilly]

imageProvider: An object that retrieves data corresponding to a representative image for the URL.

https://developer.apple.com/documentation/linkpresentation/lplinkmetadata/3143309-imageprovider?language=objc

[inPhilly]

LPLinkMetadata imageProvider is what we are looking at.

[dreampiggy]

imageProvider: An object that retrieves data corresponding to a representative image for the URL.

https://developer.apple.com/documentation/linkpresentation/lplinkmetadata/3143309-imageprovider?language=objc

If so, maybe you don't need any UTI type check, right ? It's guaranteed to be a compressed NSData of image. Can you try if this API works ?

https://developer.apple.com/documentation/foundation/nsitemprovider/2888336-loadobject

Provide the Class of UIImage, since UIImage conforms to NSItemProviderReading protocol as well...

Or, can we use the common UTI like kUTTypeImage to grab data ( you don't need to care what format it is, right ? Just need Data). And then feed the Data to our decoder (Or UIImage sd_imageWithData API)

[inPhilly]

That would probably work. I am not the most advanced coder.

[dreampiggy]

Or just wait me tomorrow. I'll watch WWDC about this and test for demos. This may need some time ;)

[inPhilly]

If that works, I still would need a custom loader for SDWebImage, correct?

[inPhilly]

Thanks. I have an iOS 13 update that is way overdue and extremely time sensitive at this point, and I am desperate to get this to work. Thank you. I will let you know if your suggestions above work to at least get the NSData.

[inPhilly]

The LPLinkMetadata is fetched separately and stored in NSUserDefaults (or shared defaults), by the way. That is a whole other issue. I am approaching this as though we already have LPLinkMetadata by the time we load (or reload) the UICollectionViewCell.

[dreampiggy]

If that works, I still would need a custom loader for SDWebImage, correct?

If that works, we can just create a simple convenient API wrapper. This is simple, compared to figure out How to generate Data.

And YES. We can provide a official Pod for this task. You can wrap your own as well if you can't wait. custom loader is designed to be public (we don't hard-coded logic related to SDWebImageDownloader) for user's use case.

[dreampiggy]

The LPLinkMetadata is fetched separately and stored in NSUserDefaults (or shared defaults), by the way. That is a whole other issue. I am approaching this as though we already have LPLinkMetadata by the time we load (or reload) the UICollectionViewCell.

See Photos Plugin that API design, we supports to provide a local identifier (we query PHAsset), or exist PHAsset (user queried by themselves). They both works. I don't think this is hard to support both (we query URL to get LPMetadata and then go on, or you provide exist LPMetadata and let me fetch image). Just API wrapper, take it easy ;)

[inPhilly]

Great. I use Carthage by the way. Ultimately, I am hoping to have these two methods available to use for UIImageView: - (void)sd_setImageWithNSItemProvider:(nullable NSItemProvider *)itemProvider placeholderImage:(nullable UIImage *)placeholder completed:(nullable SDExternalCompletionBlock)completedBlock - (void)sd_cancelCurrentImageLoad

[inPhilly]

kUTTypeImage worked perfectly.

[inPhilly]

Hello. You probably didn't get a chance to work on this, which I completely understand. I was just checking in to see. If I understood how to customize the custom loader I would do it in a heartbeat, I just don't understand quite how to do it.

[dreampiggy]

@inPhilly Sorry for this. I'm busy in my job yesterday, and for open source work, doing something related to SwiftUI.

For custom loader, here is our wiki: https://github.com/SDWebImage/SDWebImage/wiki/Advanced-Usage#custom-loader-50

You just need to implements 3 methods:

• canRequestURL (check if the URL contains the LPMetadata, or anything you want)
• requestURL (call LPMetadataProvider startFetchingMetadataForURL to grab LPMetadata if available. Then another async method to call NSItemProvider loadDataRepresentationForTypeIdentifier:. After you load the data, call our decoding logic SDImageLoaderDecodeImageData in a global queue)
• shouldBlockFailedURL: Whether or not to block failed URL, if it's indeed invalid. For example, 404 for HTTP; Or something internal Error (un-recoverable) for LinkPresentation. If nothing, just return.

You can also have a check to see how our SDWebImagePhotosPlugin works for this. Not hard logic.

[inPhilly]

So instead of PHImageRequestID requestID; I would have a property for LPMetadataProvider *linkMetadataProvider; and a property for NSProgress *itemProviderImageLoader; right? Hold onto each of these so that they can be canceled during this cancel method before being set to nil? (because each of these has their own cancel method)

[dreampiggy]

@inPhilly If you want to support cancel, you must bridge the Apple's API, to SDWebImage protocol.

PHImageRequestID is used for PHImageManager to cancel. LPMetadataProvider's API design, does not have anything like this (a token or something for cancelling). If you want to support cancel (or not, if you want it still load). You can bind each URL, a standalone LPMetadataProvider (not shared, each URL, each Provider). And just call cancel.

Today I have some time to figure this out. Please give me some times.

[dreampiggy]

I test that you don't need image data. Using the API to get UIImage from system, just simple:

After my local test, these two APIs:

• loadDataRepresentationForTypeIdentifier:
• loadObjectOfClass:

The loadObjectOfClass can be 5x~10x, faster than the Data representation one. Which means, Apple internally, already have a UIImage object, but not a NSData object. So, by default we will prefer to use loadObjectOfClass instead.

• loadDataRepresentationForTypeIdentifier benchmark

• loadObjectOfClass benchmark

[inPhilly]

A unique LPMetadataProvider is used for each request. This provider's cancel method "invokes the completion handler with the error code LPErrorMetadataFetchCancelled if the request hasn’t already completed." NSItemProvider's loadDataRepresentationForTypeIdentifier method returns an NSProgress object. The cancel method can also be called on this object.

[inPhilly]

Great work on loadObjectOfClass!

[inPhilly]

So it seems from my comment above that I have two processes that I can cancel. Now I just need to figure it all out with object ownership (weakself/strongself) and main/background threading, since I have found that LPMetadataProvider's methods need to be called on the main queue.

[dreampiggy]

I’m already been working on one implementation. You can have a check at SDWebImageLinkPlugin

[inPhilly]

At first glance that looks great. The only thing I can see so far that might need change is that a LPMetadataProvider should be allocated then assigned as a property. The cancel method is called directly on the LPMetadataProvider itself. Each LPMetadataProvider should only have one request.

[inPhilly]

Also looks at first glance like startFetchingMetadataForURL is running on the main queue, which is correct.

[inPhilly]

Note that we can also set timeout for LPMetadataProvider to less than default 30.0. Should it be left at default or set specifically?

LPMetadataProvider *lpMetadataProvider = [[LPMetadataProvider alloc] init];
lpMetadataProvider.timeout = 10;

[inPhilly]

I really do think the option to cancel is extremely important. In my particular use case this will be used to set the image on a UICollectionViewCell's UIImageView, and there are major cell reuse issues if this can't be cancelled in UICollectionViewCell's prepareForReuse.

[dreampiggy]

At first glance that looks great. The only thing I can see so far that might need change is that a LPMetadataProvider should be allocated then assigned as a property. The cancel method is called directly on the LPMetadataProvider itself. Each LPMetadataProvider should only have one request.

Emm. This is what it works. Have anything different between current code and this one below (using a wrapper instead ?)

// All the `id<SDWebImageOperation>` is retained by UIView, until finished. See `UIView+WebCacheOperation`
@interface SDWebImageMyOperation  : NSObject <SDWebImageOperation>
@proeprty (strong) LPMetadataProvider *provider;
@end

[inPhilly]

I'm not saying it doesn't work - I haven't tried it yet. Still working on reviewing everything.

My thought was, with the current code - will it call cancel on LPMetadataProvider when I call sd_cancelCurrentImageLoad on my UIImageView?

That is why I suggested. But I really am a novice developer, much below your level. So I may easily be wrong.

[dreampiggy]

I'm now a little curious about the Usage from Apple's WWDC 262. My aim of that SDWebImageLinkPlugin supports features including:

• Query image (or fallback to icon) on rich link URL, even it's not a image URL (like Apple's official site)
• View Category to support UIImageView/LPLinkView to automaically request URL
• Query exist LPLinkMetadata as well, from user's exist code
• (How to ??) Allow user to simply specified activityViewControllerLinkMetadata to Prefetch Metadata for iOS 13 share activity ? Did that means, I need to write the image onto disk, and using the NSItemProvider.contentsOf API ? (Like stupid usage, I've already have one UIImage in memory)

[dreampiggy]

I'm not saying it doesn't work - I haven't tried it yet. Still working on reviewing everything.

My thought was, with the current code - will it call cancel on LPMetadataProvider when I call sd_cancelCurrentImageLoad on my UIImageView?

That is why I suggested. But I really am a novice developer, much below your level. So I may easily be wrong.

Don't worry. Anything that have a cancel method can be retruned from that requestImageWithURL API. And it's works on cancellable from your exist SDWebImage behavior (cell-refresh and cancel previous, worked)

Actually user don't need to care about the type is. SDWebImageDownloader, return a SDWebImageDownloadToken to usage, because user may use token to check URLResponse. For link representation, currently I can not find anything reasion to create a new internal class, but actually this is small problem...

[inPhilly]

I'm now a little curious about the Usage from Apple's WWDC 262. My aim of that SDWebImageLinkPlugin supports features including:

• Query image (or fallback to icon) on rich link URL, even it's not a image URL (like Apple's official site)
• View Category to support UIImageView/LPLinkView to automaically request URL
• Query exist LPLinkMetadata as well, from user's exist code
• (How to ??) Allow user to simply specified activityViewControllerLinkMetadata to Prefetch Metadata for iOS 13 share activity ? Did that means, I need to write the image onto disk, and using the NSItemProvider.contentsOf API ? (Like stupid usage, I've already have one UIImage in memory)

Sounds fantastic. Also, I personally need the title from the LPLinkMetadata; so somewhere in the process, it might be a good idea to store it to NSUserDefaults because that is what was suggested by Apple (conforms to NSCoding).

+ (void)storeMetadata:(LPLinkMetadata*)linkMetadata forURLString:(NSString*)urlString synchronize:(BOOL)synchronize  API_AVAILABLE(ios(13.0)) {

    NSData *linkMetadataArchived = [NSKeyedArchiver archivedDataWithRootObject:linkMetadata requiringSecureCoding:YES error:nil];
    NSMutableDictionary *linkMetadatas = [[LINK_PRESENTATION_METADATA_DEFAULTS objectForKey:LINK_PRESENTATION_METADATA_DICT_NAME] mutableCopy];
    if (linkMetadatas == nil) {
        linkMetadatas = [[NSMutableDictionary alloc] init];
    }

    linkMetadatas[urlString] = linkMetadataArchived;
    [LINK_PRESENTATION_METADATA_DEFAULTS setObject:linkMetadatas forKey:LINK_PRESENTATION_METADATA_DICT_NAME];
    if (synchronize) {
        [LINK_PRESENTATION_METADATA_DEFAULTS synchronize];
    }
}

+ (LPLinkMetadata *)metadataForURLString:(NSString*)urlString  API_AVAILABLE(ios(13.0)) {

    LPLinkMetadata *linkMetadata = nil;
    NSMutableDictionary *linkMetadatas = [[LINK_PRESENTATION_METADATA_DEFAULTS objectForKey:LINK_PRESENTATION_METADATA_DICT_NAME] mutableCopy];
    if (linkMetadatas != nil) {
        NSData *linkMetadataArchived = linkMetadatas[urlString];
        if (linkMetadataArchived != nil) {
            linkMetadata = [NSKeyedUnarchiver unarchivedObjectOfClass:LPLinkMetadata.class fromData:linkMetadataArchived error:nil];
        }
    }
    return linkMetadata;
}

+ (void)removeMetadataForURLString:(NSString*)urlString synchronize:(BOOL)synchronize API_AVAILABLE(ios(13.0)) {

    NSMutableDictionary *linkMetadatas = [[LINK_PRESENTATION_METADATA_DEFAULTS objectForKey:LINK_PRESENTATION_METADATA_DICT_NAME] mutableCopy];
    if (linkMetadatas != nil) {
        if ([linkMetadatas objectForKey:urlString] != nil) {
            [linkMetadatas removeObjectForKey:urlString];
            if (synchronize) {
                [LINK_PRESENTATION_METADATA_DEFAULTS synchronize];
            }
        }
    }
}

[dreampiggy]

Another interesting behavior during my test about LinkPresentation. This API:

// Above is a LPLinkView
NSURL *url1 = [NSURL URLWithString:@"https://www.apple.com/iphone/"];
NSURL *url2 = [NSURL URLWithString:@"https://webkit.org/"];
self.linkView = [[LPLinkView alloc] initWithURL:url1];
self.linkView.frame = CGRectMake(0, 0, self.view.bounds.size.width, self.view.bounds.size.height / 2);
[self.view addSubview:self.linkView];

// Below is a UIImageView
self.imageView = [[UIImageView alloc] init];
self.imageView.frame = CGRectMake(0, self.view.bounds.size.height / 2, self.view.bounds.size.width, self.view.bounds.size.height / 2);
[self.view addSubview:self.imageView];
[self.imageView sd_setImageWithURL:url2 completed:^(UIImage * _Nullable image, NSError * _Nullable error, SDImageCacheType cacheType, NSURL * _Nullable imageURL) {
    NSLog(@"%@", @"LPMetadata image load success");
}];

Seems this does not works as my expected. Because LPLinkView does not query the image URL (such as call to startFetchingMetadataForURL). It just show a empty placeholder... 😢

[inPhilly]

Don't worry. Anything that have a cancel method can be retruned from that requestImageWithURL API. And it's works on cancellable from your exist SDWebImage behavior (cell-refresh and cancel previous, worked)

So, even though you don't call cancel on LPMetadataProvider directly in your code, it's request will still be canceled when I call sd_cancelCurrentImageLoad on my UIImageView?