Support new direct_messages/events API

[collinsauve]

The old DM API is deprecated and the migration guide suggest that:

Developers should migrate to the relative API endpoint as described below by August 16th, 2018.

This is a very significant rewrite of the DM API. Main changes to be aware of:

• Pagination is achieved via cursors rather than since_id and max_id
• This combines both sent and received message
• Only 30 days worth of events can be retreived
• The response events no longer contain the screenname, only the user id. This makes it difficult to determine who is sending and who is receiving the message if consumer is used to working with screennames. (thread on the forum)

Additionally, based on the documentation provided, I can't tell what events.message_create.message_data.entities.symbols includes.

[ahmad-atallah]

As Mr. Yortw stated, he's not planning to update TweetMoaSharp to include the new Twitter API. I've been using TweetMoaSharp and it's the easiest library I saw to connect to twitter.. what is the replacement now? Twitter will stop the old API in August!!

[collinsauve]

As Mr. Yortw stated, he's not planning to update TweetMoaSharp to include the new Twitter API.

I haven't seen that statement, can you provide a reference?

[ahmad-atallah]

Sorry got busy, I asked him on Twitter.. Here

[johnkamau]

I assume we are on our own regarding the new changes to the api? or is anyone working on something?

[ahmad-atallah]

We have 10 days left :\ I'm torn between something called Tweetinvi and LinqToTwitter.. but I just love how TweetMoaSharp is so simple to use and very efficient

[johnkamau]

Tweetinvi suffers from the same problem last I checked. Still uses the old api for DMs. I guess we may have to implement our own tiny changes specifically for DMs since the other apis remain unchanged.

[ahmad-atallah]

If you found something or reached a solution, it would be highly appreciated to share !

[johnkamau]

LinqToTwitter seems to have been updated. Want to try using it just for DMs and leave the rest to TweetMoaSharp for now as a quick fix.

[ahmad-atallah]

LOL! That's exactly what I (almost) decided to do.. This question could be useful for you if you want to go on with LinqToTwitter.

[johnkamau]

Ah.. thanks for the link to that stack overflow question, was actually wondering how to get started with LinqToTwitter.

[ahmad-atallah]

@johnkamau were you able to achieve anything with LinqToTwitter?

[johnkamau]

Using the examples on SO, I've managed to get the DMs and get it to kind of work how TweetMoaSharp works. The main issue I faced was keeping track of the newest DM since I last fetched the list. This forced me to keep a local cache of the last DMs I got, and use the ID from the last item I got to count how many new ones have arrived.

Since the direct messages api has a limit of 15 per user per 15 minutes, it meant I limited my polling to once every 3 minutes.

Also, since the new api doesn't have the sender details, I was forced to get use the /lookup api to get the sender details for any DMs I receive (if new).

[ahmad-atallah]

@johnkamau Do you think you can help me through this? I'm still not able to get the DMs at all :/

[johnkamau]

My code is in vb.net, will that be helpful?

[ahmad-atallah]

That would be wonderful!! My email is visible in my profile

[johnkamau]

Here's an example of what I got to work, it's pretty rough and should be optimised better, but for now it kinda works.

Dim CredStore As New SessionStateCredentialStore(HttpContext.Current.Session)
Dim Token As OAuthRequestToken = HttpContext.Current.Session(GlobalSessionTwitterToken)
Dim TwitterContext As TwitterContext = Nothing
Dim GlobalTwitterPageSize = 50

If Token IsNot Nothing Then
    CredStore.ConsumerKey = __ConsumerKey 'I assume you have the key, secret, token, and token secret somewhere
    CredStore.ConsumerSecret = __ConsumerSecret
    CredStore.OAuthToken = Token.Token
    CredStore.OAuthTokenSecret = Token.TokenSecret
    Dim Auth As New AspNetAuthorizer()
    Auth.CredentialStore = CredStore
    TwitterContext = New TwitterContext(Auth)
else
    'Redirect them here to get your token and store it in session
End If

'From here this is pretty much a copy paste from the Linq2Twitter examples
Dim AllDmEvents As List(Of DMEvent) = New List(Of DMEvent)()
Dim DmResponse As DirectMessageEvents = Await (From dm In TwitterContext.DirectMessageEvents Where dm.Type = DirectMessageEventsType.List AndAlso dm.Count = GlobalTwitterPageSize Select dm).SingleOrDefaultAsync()
Dim Cursor As String = ""

AllDmEvents.AddRange(DmResponse.Value.DMEvents)
Cursor = DmResponse.Value.NextCursor

'This gets all the DMs from all the pages
While Not String.IsNullOrWhiteSpace(Cursor)
    DmResponse = Await (From dm In TwitterContext.DirectMessageEvents Where dm.Type = DirectMessageEventsType.List AndAlso dm.Count = GlobalTwitterPageSize AndAlso dm.Cursor = Cursor Select dm).SingleOrDefaultAsync()
    AllDmEvents.AddRange(DmResponse.Value.DMEvents)
    Cursor = DmResponse.Value.NextCursor
End While

Dim Since as Long = 0 'From your cache, get the last DM you saved

Dim SenderIDs As New List(Of Integer) 'List of IDs to get the sender details
Dim NewCount As Integer
Dim DmEventsToLookUp as new List(Of DMEvent) 'We only want to lookup new entities so we log them here

For Each DmEvent In AllDmEvents
    If CLng(DmEvent.ID) > Since Then
        SenderIDs.Add(CLng(DmEvent.MessageCreate.SenderID))
        NewCount += 1
        DmEventsToLookUp.Add(DmEvent)
    End If
Next

If DmEventsToLookUp.Count<=0
    Exit Function 'There are no new Dms to lookup
End If

Dim UniqueSenderIDsCommaSeparated = String.Join(",", SenderIDs)
Dim UsersListResponse = Await (From TwUser In TwitterContext.User Where TwUser.Type = UserType.Lookup AndAlso TwUser.UserIdList = UniqueSenderIDsCommaSeparated Select TwUser).ToListAsync()
Dim NewSince As Long = AllDmEvents.First().ID 'Save this to your cache as the latest ID

For Each DmEvent In DmEventsToLookUp
    'Process your DMs here, and save to your cache
    MyCustomObject.Sender = UsersListResponse.FirstOrDefault(Function(f) f.UserIDResponse = DmEvent.MessageCreate.SenderID) 'Get your sender details here
Next

[ahmad-atallah]

@johnkamau Thank you so much!! I've been trying all this time to get it work, but no luck yet :( When debugging, after I reach this

DirectMessageEvents dmResponse =
            await
                (from dm in twitterCtx.DirectMessageEvents
                 where dm.Type == DirectMessageEventsType.List &&
                 dm.Count == 10
                 select dm)
                .SingleOrDefaultAsync();

it just go away and never come back!! Also, I have a problem with async and await ! How I can call these methods in button click?

[johnkamau]

Use of await and async is a bit of a large topic. A simple solution is to wrap the code in an async function that returns a Task(Of Result) (where result can be a string, list etc), then in the calling code call MyFunc().Result. The result object should have the output of the async function.

[ahmad-atallah]

Can you please have a look at what I tried and tell me what's wrong?

protected void Btn1_Click(object sender, EventArgs e)
    {
        string x = mytest().Result;
    }

and mytest is:

static async Task<string> mytest()
    {
        AspNetAuthorizer auth = DoAuthorization();
        var twitterCtx = new TwitterContext(auth);

        List<DMEvent> AllDmEvents = new List<DMEvent>();
        string Cursor;
        DirectMessageEvents dmResponse =
            await
                (from dm in twitterCtx.DirectMessageEvents
                 where dm.Type == DirectMessageEventsType.List &&
                 dm.Count == 10
                 select dm)
                .SingleOrDefaultAsync(); //In debugging mode, after this line is executed, it will go away and keep loading forever and never come back

        AllDmEvents.AddRange(dmResponse.Value.DMEvents);
        Cursor = dmResponse.Value.NextCursor;

        string xxx = (JsonConvert.SerializeObject(AllDmEvents, Formatting.None));
        return xxx;
    }

DoAuthorization is:
  static AspNetAuthorizer DoAuthorization()
    {
        AspNetAuthorizer auth = new AspNetAuthorizer();

        auth = new AspNetAuthorizer
        {
            CredentialStore = new SessionStateCredentialStore
            {

                ConsumerKey = "ConsumerKey",
                ConsumerSecret = "ConsumerSecret",
                OAuthToken = "OAuthToken",
                OAuthTokenSecret = "OAuthTokenSecret",
                ScreenName = "MyScreenName",
                UserID = 123456789
            }
        };

        return auth;
    }

[johnkamau]

I am not that familiar with C#, see results from Stack Overflow

https://stackoverflow.com/questions/45458178/why-does-await-never-return

[ahmad-atallah]

I'm gonna try to translate my code to VB: Button click:

Protected Sub Btn1_Click(ByVal sender As Object, ByVal e As EventArgs)
        Dim x As String = mytest.Result
    End Sub

and mytest:

Private Shared Async Function mytest() As Task(Of String)
        Dim auth As AspNetAuthorizer = DoAuthorization
        Dim twitterCtx = New TwitterContext(auth)
        Dim AllDmEvents As List(Of DMEvent) = New List(Of DMEvent)
        Dim Cursor As String

Dim DmResponse As DirectMessageEvents = Await (From dm In twitterCtx.DirectMessageEvents Where dm.Type = DirectMessageEventsType.List AndAlso dm.Count = 10 Select dm).SingleOrDefaultAsync()

        'In debugging mode, after this line is executed, it will go away and keep loading forever and never come back
        AllDmEvents.AddRange(dmResponse.Value.DMEvents)
        Cursor = dmResponse.Value.NextCursor
        Dim xxx As String = JsonConvert.SerializeObject(AllDmEvents, Formatting.None) //this just to serialize the result to json
        Return xxx
    End Function

And finally, DoAuthorization:

 Private Shared Function DoAuthorization() As AspNetAuthorizer
        Dim auth As AspNetAuthorizer = New AspNetAuthorizer
        Dim CredStore As New SessionStateCredentialStore(HttpContext.Current.Session)

        CredStore.ConsumerKey = "ConsumerKey"
        CredStore.ConsumerSecret = "ConsumerSecret"
        CredStore.OAuthToken = "OAuthToken"
        CredStore.OAuthTokenSecret = "OAuthTokenSecret"

        auth.CredentialStore = CredStore

        Return auth
    End Function

[johnkamau]

The last entry in the link below explains why the call is hanging I think.

https://stackoverflow.com/questions/14526377/why-does-this-async-action-hang

[collinsauve]

Guys I sympathize with your situation but this is not the place for that kind of in depth discussion about a different library.

On Thu, Aug 9, 2018, 9:43 AM John Kamau, notifications@github.com wrote:

The last entry in the link below explains why the call is hanging I think.

https://stackoverflow.com/questions/14526377/why-does-this-async-action-hang

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Yortw/tweetmoasharp/issues/67#issuecomment-411761462, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJno4C9kZA25hBZKIjZRPeDGJqRGVx-ks5uPDxqgaJpZM4UdMg_ .

[johnkamau]

@collinsauve I agree. Apologies.

[Yortw]

Hi everyone;

The good news is I have started work on a new major release (5.0). The bad news is it is slow going and I can make no promises about either when it will be ready or what it will look like. That said I could do with some feedback from anyone with a vested interest in a new TweetMoaSharp release.

As part of this release I plan to;

1. Drop support for the streaming API
2. Drop support for the old direct messages API
3. Add support for the new direct messages API

4. 69 Drop and add support for various platforms

5. 70 Upgrade the Json.Net dependency version

For the last two, please leave comments/thoughts on those issues. For the first two, I'm open to discussion if anyone thinks there's a real reason to keep these, but as Twitter is in the process of killing them entirely I don't see any point in continuing to support them.

That leaves item 3, which is best discussed here.

One issue with implementing the new API is what it should look like in TMS. The older Twitter API's all use url query strings to pass arguments to the API (with a few exceptions like uploading media). The new API requires a json body in a specific format, and converting the arguments passed to the TMS method into the json required can be done several ways.

If we keep things consistent with the rest of TMS, a call to send a direct message would look something like;

    var result = twitterService.SendDirectMessage(new DirectMessageOptions() { Text = "Hello!", recipientId = 3232412, mediaType = "dm_gif", mediaId = 234234242342,  quickReplies = new QuickReply[] { new QuickReply() { Id = "1", Label = "Hi!", Description = "Hello" }, new QuickReply() { Id = "2", Label="Go away", Description = "Busy"} });

The current TMS 'DSL' (such that it is) will generate the options class used, but we will still need something to format this into the required json body. I'm not going to have time to update the DSL and write all the code generation logic to generate those classes, so I'll probably hand code them for now. To be honest, once I start doing that, unless there is a community effort to update the DSL to cope, I suspect the hand coding will live on as the plan for all future API work.

An alternative would be to have the TMS call look more like the Twitter API itself;

    var result = twitterService.SendDirectMessage(
       new DirectMessageOptions() 
       {
          MessageCreateEvent = new TwitterMessageCreateEvent()
          {
            MessageCreate = new TwitterMessageCreate()
            {
               Target = new TwitterMessageTarget()
               {
                  recipientId = 1231313213
               },
              MessageData = new TwitterMessageData()
              {
                Text = "Hello",
                Attachment = new TwitterMessageAttachment()
                {
                   MediaId = 234243242
                },
                QuickReplies = new QuickReply[]
                {
                   new QuickReply() { Id = "1", Label = "Hi", Description = "Hello" },
                   new QuickReply() { Id = "2", Label = "Go away", Description = "Busy" }
                }
              }
            }
          }
       }
   ); 

The advantage of this is that the shape of the arguments can be used to generate the required request body just by passing the object graph to Json.Net (assuming some appropriate attributes on the classes). However having just written that out by hand and looking at it, it now seems obviously terrible so I think plan one is the way to go.

I guess I've just answered my own question here. Feedback on the other issues still appreciated.

[Yortw]

There is a pre-release package (5.0.0-beta) available on Nuget (https://www.nuget.org/packages/TweetMoaSharp/5.0.0-beta) if anyone wants to try it and submit feedback before I pull the trigger on an official release.

The migration guide here is likely to be very useful; https://developer.twitter.com/content/dam/developer-twitter/pdfs-and-files/DM - Migration Guide.pdf

The new API sucks, so don't expect magic or pleasantness, I have kept to the principle of 'thin and light weight wrapper around the Twitter API' so this exposes most of their ugliness, and means the Twitter advice/guides are the best 'how to' source.

[Yortw]

Supported in v5.0.0.