Issue with accessing unknownFields

[dayanch96]

Hey! Hope you're doing well

Starting with version 19.35.3, YouTube no longer uses GPBMessage *unknownFields directly. Instead, I now see the ivar NSMutableData *unknownFieldData_. Any attempt to use unknownFields results in an application crash as selector is unrecognized.

By the way, it probably should be created in separated report, but you're using youtube.com/watch?=contentID for Shorts instead of youtube.com/shorts/contentID

[jkhsjdhjs]

Hey, nice hearing from you again! I'm doing pretty good, hope you as well!

I hadn't updated YouTube yet, but now I also see this issue. Seems like there have been a lot of deprecations in the UnknownFields API:

Tweak.x:92:5: error: 'GPBUnknownFieldSet' is deprecated: Use GPBUnknownFields instead. [-Werror,-Wdeprecated-declarations]
    GPBUnknownFieldSet *fields = shareEntity.unknownFields;
    ^~~~~~~~~~~~~~~~~~
    GPBUnknownFields
protobuf/objectivec/GPBUnknownFieldSet.h:18:16: note: 'GPBUnknownFieldSet' has been explicitly marked deprecated here
__attribute__((deprecated("Use GPBUnknownFields instead.", "GPBUnknownFields")))
               ^
Tweak.x:92:46: error: 'unknownFields' is deprecated: Use GPBUnknownFields and the -initFromMessage: initializer and mergeUnknownFields:extensionRegistry:error: to add the data back to a message. [-Werror,-Wdeprecated-declarations]
    GPBUnknownFieldSet *fields = shareEntity.unknownFields;
                                             ^
protobuf/objectivec/GPBMessage.h:77:5: note: 'unknownFields' has been explicitly marked deprecated here
    deprecated("Use GPBUnknownFields and the -initFromMessage: initializer and "
    ^

So we could definitely adjust the code to use the new API and it would work again. However, I wonder if it would be cleaner to parse the serializedShareEntity using a custom Protobuf Message that has the correct fields for easier access. This would allow simply using protobuf_message.video or protobuf_message.playlist instead of extracting each field individually via the UnknownFields API the way it's currently done.

One way of doing this would be using a predefined Protobuf Message from the YouTube app to parse the serializedShareEntity. However, I don't think the YouTube app actually has such a class, I assume it doesn't interact with the serializedShareEntity at all and simply receives it from the server alongside the content and only sends it back to the server when sharing something. The YouTube web frontend also sends a serializedShareEntity when sharing a video, this further strengthens my belief that this is actually the case and the frontends/apps don't serialize the share entity themselves.

The other option would be implementing a protobuf message for the share entity in the tweak and using that to deserialize the share entity. So this is what it would look like:

syntax = "proto3";

message Clip {
    string id = 1;
}

message ShareEntity {
    string video = 1;
    string playlist = 2;
    string channel = 3;
    string post = 6;
    Clip clip = 8;
    optional int32 unknown_short_value = 20;
}

I used this to generate Objective C files and included them. It compiles, but fails during linking because of undefined symbols:

==> Linking tweak YouTubeNativeShare (arm64)…
Undefined symbols for architecture arm64:
  "_OBJC_CLASS_$_GPBMessage", referenced from:
      _OBJC_CLASS_$_Clip in ShareEntity.pbobjc.x.29d40fd5.o
      _OBJC_CLASS_$_ShareEntity in ShareEntity.pbobjc.x.29d40fd5.o
  "_OBJC_CLASS_$_GPBRootObject", referenced from:
      _OBJC_CLASS_$_ShareEntityRoot in ShareEntity.pbobjc.x.29d40fd5.o
  "_OBJC_METACLASS_$_GPBMessage", referenced from:
      _OBJC_METACLASS_$_Clip in ShareEntity.pbobjc.x.29d40fd5.o
      _OBJC_METACLASS_$_ShareEntity in ShareEntity.pbobjc.x.29d40fd5.o
  "_OBJC_METACLASS_$_GPBRootObject", referenced from:
      _OBJC_METACLASS_$_ShareEntityRoot in ShareEntity.pbobjc.x.29d40fd5.o

The undefined symbol errors make sense, as I'm not building protobuf in its entirety, but just the generated files. To resolve them, I would either have to

• build protobuf as well, or
• resolve the symbols dynamically at runtime in the YouTube app.

I tried building Protobuf but encountered some errors. It is definitely possible, since YouTube does it as well, but it would increase the size of the tweak a lot and thus wouldn't be worth it IMO.

I'm not sure whether it's possible to resolve these symbols dynamically at runtime, but theos does so as well when using %c(SomeClass), so shouldn't it be possible here as well somehow?

Finally, regarding the links created for shorts: The unknown_short_value shown in the protobuf message above is only present when sharing a short, so it can be used to differentiate between shorts and normal videos and thus use /shorts/contentID when sharing a short. This can definitely be implemented once the other stuff is fixed.

[dayanch96]

Thank you for the detailed reply

theos does so as well when using %c(SomeClass), so shouldn't it be possible here as well somehow?

%c(SomeClass) is just a macro, and the linker can't resolve it if the class hasn't been declared. You can use NSClassFromString(@"SomeClass"), which avoids linker issues, but the class still needs to be properly declared and implemented for everything to work correctly.

Could this be uploaded to a separate branch? I might be able to help, even if just a little

[jkhsjdhjs]

Sure, I pushed it on the protobuf_testing branch.

I adjusted the Makefile to automatically generate the protobuf files from share-entity.proto during the build. However, this currently expects protoc to be installed.

The build currently yields the following linker errors:

  "_GPBCheckRuntimeVersionSupport", referenced from:
      _GPB_DEBUG_CHECK_RUNTIME_VERSIONS in ShareEntity.pbobjc.m.29d40fd5.o
  "_OBJC_CLASS_$_GPBDescriptor", referenced from:
      objc-class-ref in ShareEntity.pbobjc.m.29d40fd5.o
  "_OBJC_CLASS_$_GPBMessage", referenced from:
      _OBJC_CLASS_$_Clip in ShareEntity.pbobjc.m.29d40fd5.o
      _OBJC_CLASS_$_ShareEntity in ShareEntity.pbobjc.m.29d40fd5.o
  "_OBJC_CLASS_$_GPBRootObject", referenced from:
      _OBJC_CLASS_$_ShareEntityRoot in ShareEntity.pbobjc.m.29d40fd5.o
  "_OBJC_METACLASS_$_GPBMessage", referenced from:
      _OBJC_METACLASS_$_Clip in ShareEntity.pbobjc.m.29d40fd5.o
      _OBJC_METACLASS_$_ShareEntity in ShareEntity.pbobjc.m.29d40fd5.o
  "_OBJC_METACLASS_$_GPBRootObject", referenced from:
      _OBJC_METACLASS_$_ShareEntityRoot in ShareEntity.pbobjc.m.29d40fd5.o

The first one can be fixed by commenting out GPB_DEBUG_CHECK_RUNTIME_VERSIONS(), given the same I assume it's just some safety check, which isn't required.

The _OBJC_CLASS_$_GPBDescriptor error can be resolved by replacing the reference to GPBDescriptor with %c(GPBDescriptor). This required manually editing ShareEntity.pbobjc.m and renaming it to .x instead of .m, so logos macros are applied. The Makefile also needs to be adjusted to reflect this name change.

This leaves us with 4 linker errors. Thanks for giving it a try in advance, would be cool if we could get this to work :D

[dayanch96]

Awesome! I just woke up and was checking if there any news. Be back very soon. Have a good one

[dayanch96]

Sure, I pushed it on the protobuf_testing branch.

I adjusted the Makefile to automatically generate the protobuf files from share-entity.proto during the build. However, this currently expects protoc to be installed.

The build currently yields the following linker errors:

  "_GPBCheckRuntimeVersionSupport", referenced from:
      _GPB_DEBUG_CHECK_RUNTIME_VERSIONS in ShareEntity.pbobjc.m.29d40fd5.o
  "_OBJC_CLASS_$_GPBDescriptor", referenced from:
      objc-class-ref in ShareEntity.pbobjc.m.29d40fd5.o
  "_OBJC_CLASS_$_GPBMessage", referenced from:
      _OBJC_CLASS_$_Clip in ShareEntity.pbobjc.m.29d40fd5.o
      _OBJC_CLASS_$_ShareEntity in ShareEntity.pbobjc.m.29d40fd5.o
  "_OBJC_CLASS_$_GPBRootObject", referenced from:
      _OBJC_CLASS_$_ShareEntityRoot in ShareEntity.pbobjc.m.29d40fd5.o
  "_OBJC_METACLASS_$_GPBMessage", referenced from:
      _OBJC_METACLASS_$_Clip in ShareEntity.pbobjc.m.29d40fd5.o
      _OBJC_METACLASS_$_ShareEntity in ShareEntity.pbobjc.m.29d40fd5.o
  "_OBJC_METACLASS_$_GPBRootObject", referenced from:
      _OBJC_METACLASS_$_ShareEntityRoot in ShareEntity.pbobjc.m.29d40fd5.o

The first one can be fixed by commenting out GPB_DEBUG_CHECK_RUNTIME_VERSIONS(), given the same I assume it's just some safety check, which isn't required.

The _OBJC_CLASS_$_GPBDescriptor error can be resolved by replacing the reference to GPBDescriptor with %c(GPBDescriptor). This required manually editing ShareEntity.pbobjc.m and renaming it to .x instead of .m, so logos macros are applied. The Makefile also needs to be adjusted to reflect this name change.

This leaves us with 4 linker errors. Thanks for giving it a try in advance, would be cool if we could get this to work :D

Isn't this file auto-generated?

Guess linker issue in your Makefile: YouTubeNativeShare_LIBRARIES = protobuf

I might be wrong, but as far as i know $(TWEAK_NAME)_LIBRARIES will look for the library in $(THEOS)/lib directory

One more thing that might be useful. You're adding the entire YouTubeHeader repository to your project, but only using topViewControllerForPresenting from YTUIUtils. Wouldn't it be simpler to declare just what we need?

@interface YTUIUtils : NSObject
+ (UIViewController *)topViewControllerForPresenting;
@end

[jkhsjdhjs]

Isn't this file auto-generated?

Yep, but if we can get it to work with modifying the generated file, we can think about how we can get it to work without modifying it. The thing is, we could probably get it to work by also building the protobuf library, but since the YouTube app should already contain all required symbols, this shouldn't be necessary at all. Maybe we can simply define interfaces for the symbols the linker currently can't find.

Guess linker issue in your Makefile: YouTubeNativeShare_LIBRARIES = protobuf

Ah yes, I just added this to experiment a bit and forgot to remove it. Anyway, after removing it the linker issues remain.

One more thing that might be useful. You're adding the entire YouTubeHeader repository to your project, but only using topViewControllerForPresenting from YTUIUtils. Wouldn't it be simpler to declare just what we need?

Yeah, that's true. I initially added it because I planned to use more definitions available there, but noticed later that YouTubeHeader doesn't contain many interface definition I needed. So yes, you're right, it might as well be removed.

[jkhsjdhjs]

Adding -undefined dynamic_lookup to LDFLAGS seems to work around the undefined symbol issue, although it is apparently deprecated, it at least makes the build work. However, the tweak still doesn't work with it, i.e. the hooked functions aren't called at all. I noticed that the hooked functions are called when I remove proto/ShareEntity.pbobjc.m from YouTubeNativeShare_FILES in the Makefile (and also commenting out the import of its header and some other stuff. However, as soon as I only add proto/ShareEntity.pbobjc.m to the FILES array again, the hooks stop working. Not sure why that is.

[dayanch96]

Maybe we can simply define interfaces for the symbols the linker currently can't find.

That's quite possible — we can check that. I’m not very familiar with how protobuf works, but I'll try to find the necessary headers. What specifically do we need besides what is already included in Tweak.x?

the hooked functions aren't called at all

maybe problem with using -undefined dynamic_lookup?

[jkhsjdhjs]

That's quite possible — we can check that. I’m not very familiar with how protobuf works, but I'll try to find the necessary headers. What specifically do we need besides what is already included in Tweak.x?

Hmm, not sure, all necessary headers should already by provided by protobuf.

maybe problem with using -undefined dynamic_lookup?

I also thought so at first, but then I removed proto/ShareEntity.pbobjc.m from the Makefile and the tweak functioned as usual, i.e. the hooks were at least working. So adding proto/ShareEntity.pbobjc.m to the Makefile seems to disturb something. Do you know if there's anything special to keep in mind when adding more than one file to the Makefile?

[jkhsjdhjs]

Seems like any @implementation block breaks the hooks, even if it's just an empty block like

@implementation ShareEntityRoot

// No extensions in the file and no imports or none of the imports (direct or
// indirect) defined extensions, so no need to generate +extensionRegistry.

@end

So commenting out all @implementation blocks makes the hooks work, having even a single @implementation block breaks them. Really don't know why.

[yodaluca23]

Hey, any updates on this? Sorry if I'm being annoying, I really love this tweak. Hope the new YouTube changes can be figured out!

[dayanch96]

Hey. Sorry for late reply - been really busy these days with irl stuff.

Seems like any @implementation block breaks the hooks, even if it's just an empty block like

So commenting out all @implementation blocks makes the hooks work, having even a single @implementation block breaks them. Really don't know why.

It's unlikely because I use a lot of similar %hook and @implementation combinations in my tweaks.

The only thing is that %hook requires a .x or .xm file, as it's a Theos macro. But you can use both, and %hook and @implementation in the one file at the same time.

The problem is most likely in the Makefile settings and/or how the generated ShareEntity.pbobjc.m works

and, as YouTube trying to improve their share sheet by adding timestamp, posting and etc, I just removed si=ID as temporary workaround

static NSString *cleanShareUrl(NSString *string) {
    if (string.length > 0) {
        BOOL isCopy = [string containsString:@"?si="];
        BOOL isShare = [string containsString:@"&si="];

        if (isShare || isCopy) {
            NSArray *components = [string componentsSeparatedByString:isCopy ? @"?si=" : @"&si="];

            if (components.count == 2) {
                return components.firstObject;
            }
        }
    }

    return string;
}

%hook _UIConcretePasteboard
- (void)setString:(NSString *)shareURL {
    NSString *string = cleanShareUrl(shareURL);
    %orig(string);
}
%end

%hook YTIIosSystemShareEndpoint
- (NSString *)shareURL {
    NSString *string = %orig;
    return cleanShareUrl(string);
}
%end

[jkhsjdhjs]

I pushed a fix for this issue in b4fd811d7a5d48c436b7f2d04ce12ef8ac363b44, but I'll leave this issue open, since defining a protobuf message for this would be the cleaner solution.