Special characters in tweet's text are not being escaped properly

[annjawn]

Swifter is not returning the "full_text" from the tweet object with properly escaped special characters for &, >, < characters.

I tweeted "Ben & Jerry's" and this is what the json shows up as

.....
    "truncated" : false,
    "geo" : null,
    "full_text" : "Ben & Jerry’s",
    "in_reply_to_user_id" : null,
    "in_reply_to_user_id_str" : null
...

This is the case with client.getTimeline method but I am sure it could be the issue with all the other API methods.

[ghost]

I ran into this myself a couple months ago. As far as I can tell from perusing a number of forum posts and stackoverflow answers, this is just the way Twitter formats it’s responses, not a bug in Swifter. You’ll have to replace these yourself, though Swift makes that pretty easy with the replacingOccurrences(of:with:) method on String.

[annjawn]

yes realizing that, I ultimately ended up writing my own String extension, but it's a little more elaborate than replacingOccurrences(of:with:). However though, the Swifter json is giving me strange errors when it contains these special characters.

[ghost]

What kinds of strange errors?

[annjawn]

As you may recall from this conversation --> #152 that I am trying to get the Decodable protocol to work with swifter. I have been able to make it work for the most part of it, as shown below. HOWEVER, the statement JSONDecoder().decode fails if the full_text in the tweet JSON contains this string -/:;()$&@“.,?!’[]{}#%^*+=_\|~<>€£¥•.,?!’ , it simply says that the Data isn't in the correct format and no amount of replacingOccurrences worked. If I delete that tweet and run it again, it works just fine (user_timeline). I have finally resorted to using TwitterKit SDK and the API that it provides and that is working well for me and I can use Decodable on the Data returned by the request. I feel using Decodable to build the Tweet model is way more elegant and readable than doing a json["foo"]["bar"]["imet"]....

swifter.getUserFollowing(for: .id("16532709"), cursor: nil, count: nil, skipStatus: true, includeUserEntities: false, success: { (json, prev, next) in
     let jsonData = String(describing: json)
     let jsD   = jsonData.data(using: .utf8)!
     do{
            let json = try JSONDecoder().decode([userIDS].self, from: jsD)
          }catch{
                    print("ANN: JSON Error")
         }
      }, failure: { (error) in
                print("Error: ", error.localizedDescription)
            })

[ghost]

Oh, interesting. I’ll look into this later. My gut says that there’s some sort of problem with the way the JSON type pretty-prints strings in the description computed var, but can’t be sure without investigating.

I agree re: Decodable, there is some cost associated with creating types to decode to, but the code-completion and compile-time checking of field names, plus the ability to abstract away and/or change the decoded structure of the data, far outweigh them imo.

Did you ever try out my raw_data_and_decodable branch @ https://github.com/zachrwolfe/Swifter/tree/raw_data_and_decodable ? That should also solve your problem for now until we can get a version of that merged into master.

[sbeitzel]

Is anyone looking at this?

I'm rewriting a client app for macOS that I had previously written in Java using Twitter4J. That library solved this problem by doing entity replacement internally, since there are not only HTML entities like & but also Twitter-specific entities like media and links.

Above, TwitterKit gets mentioned, but that library is well past its end-of-life.