Stream not having any match (while it should)

[PsykotropyK]

Hi,

I am not sure if I post at the good place so please let me know if I should move this post somewhere else.

I have a little console tester application that allow me to test a piece of code before including it in my main app. Regarding a twitter stream the code look like this:

        static void Main(string[] args)
        {
            Auth.SetUserCredentials(CREDENTIALS);

            // Access the filtered stream
            var filteredStream = Stream.CreateFilteredStream();
            filteredStream.AddTrack("#ComeTogether"); //this is the twitter trending hashtag at time of execution
            filteredStream.MatchingTweetReceived += (sender, argsstr) => { received(argsstr); };
            var StreamThread = new Thread(() => filteredStream.StartStreamMatchingAllConditions());
            StreamThread.Start();                  

        }

        static void received(MatchedTweetReceivedEventArgs TweetReceived)
        {
            Console.WriteLine(TweetReceived.Tweet.Text);
        }

This works perfectly fine. But when I put it in my main app, there is no more tweet received for the same hashtag (actually a list of 13 hashtags + the current trending one to be sure to receive something). My code looks like this:

        /// <summary>
        /// RunStream will create a twitter stream in a dedicated thread to handle tweets gathering
        /// </summary>
        /// <param name="SearchCriterias">Search criterias (string criteria, int idunderlying) </param>
        public void RunStream(List<streamParameters> SearchCriterias)
        {

            if (SearchCriterias.Count > 0)
            {
                TwitSetCredentials();
                for (int i = 0; i <= SearchCriterias.Count - 1; i++)
                {
                    MainStream.AddTrack(SearchCriterias[i].StreamCriteria);
                }

                MainStream.MatchingTweetReceived += (sender, args) =>
                {
                    ReceivedTweet(args);
                };

                // TODO switch to TPL (Task Pool Library)
                Thread StreamTask = new Thread(() => MainStream.StartStreamMatchingAllConditions());
                StreamTask.Start();
            }
        }

        /// <summary>
        /// Root function of the tweet analysis algorithm
        /// </summary>
        /// <param name="TweetReceived">tweet received</param>
        public void ReceivedTweet(MatchedTweetReceivedEventArgs TweetReceived)
        {
            //DO STUFF
        }

Here, ReceivedTweet never got called.

Note that the whole thing works with AvalonDock as a window manager (in case there is some known issues) which is basically the only big third party reference used. I also controlled my console app with more than one Hashtag, and it still works fine.

Any guess on what I am doing wrong?

Any help would be greatly appreciated. Thanks in advance.

[linvi]

Hello,

There has been a new release (0.9.10.0) of Tweetinvi during the night. Would you please confirm which version of Tweetinvi you are using?

Also would you please let me know which type of project this code is not working on. And finally, do you have multiple streams running at the same time?

Cheers, Linvi

[PsykotropyK]

Hi Linvi,

I will confirme the exact version in the evening but I am using the last version available on NuGet (I tried to update before my initial post but I was up to date). I think its the 0.9.9.8 but I'll confirm later.

I am not sure what you mean by "type of project" but basically I am using AvalonDock as a window manager. At the moment of execution it is loaded with one tool window very basic (one treeview and a toggle switch button that command the stream). I also have references to Oxyplot (chart library) but nothing was loaded from that at the moment.

All the data (to construct the treeview, find the stream parameter, save the results, populate the charts if needed,....) is hosted in a sql server 2008 r2 database that I access using Entity Framework.

Regarding the number of stream: just the one mentionned. Note that I test my main app with minimal other functions called (only the loading of AvalonDock and the window used to command the stream as said earlier).

I try with a task instead of a threaf with the same lack of results. I will try without creating any of those to see if I have more luck.

Cheers

[linvi]

I was asking if the application was a WPF, Windows 10 Universal App, Windows Phone, ASP.NET... The fact that you are using Avalon suggest that you are using WPF but can you confirm this?

I will investigate this issue when I have some available time.

[PsykotropyK]

Indeed its WPF

[PsykotropyK]

Linvi, I updated to 0.9.10.0 but I still got the same issue. I will create a new WPF app and gradually build it up to something similar to what I have and test it at every step. I will communicate when it stops working.

[linvi]

Sorry for not replying earlier, it was quite a busy day. I have just verified and I have successfully run a FilteredStream in WPF.

Please keep me up to date.

[PsykotropyK]

Hi Linvi,

I may have found something.

From a console app or a "blank" WPF app :

If I don't use threading, the method I call with MatchingTweetReceived must be static and can be either public or private.

If I use a thread, it needs to be static and private (quite the opposite of what I had in my app).

On my app though I still have something that bugs.

[linvi]

Please share your code, this is very hard to understand with words!!!

[PsykotropyK]

I kept on looking and anyway I was not fully right before. So with the code:

Assume that I have a blank new WPF project with only a button on my MainWindow.xaml

        private void Button_Click(object sender, RoutedEventArgs e)
        {
             Auth.SetUserCredentials(CREDENTIALS);

            // Access the filtered stream
            var filteredStream = Stream.CreateFilteredStream();
            filteredStream.AddTrack("#JACOBWIRED2015");
            filteredStream.AddTrack("#ComeTogether");
            filteredStream.MatchingTweetReceived += (sendertwt, argsstr) => 
            { 
                received(argsstr); 
            };
            var StreamThread = new Thread(() => filteredStream.StartStreamMatchingAllConditions());
            StreamThread.Start();
            StreamThread.IsBackground = true;
        }

        private void received(MatchedTweetReceivedEventArgs TweetReceived)
        {
            //DOSTUFF
        }

received is not called if I dont have the StreamThread.IsBackground = true; line or if received is public instead of private.

On my main app, it was both without the IsBackground and with a public method. However, even if I fix this, it still does not work.

[linvi]

I would personally not do it this way. The reason is that you are creating a FilteredStream is a function that is invoked and finished right away.

As there is no external reference to the var filteredStream, the Garbage Collector will be free to collect the Stream. It will therefore result in the stream to be stopped.

Instead of doing this I would rather store the filteredStream as a private member of your class.

// Garbage Collector will not collect the _sampleStream as it is referenced by the MainWindow object
private ISampleStream _sampleStream;

public MainWindow()
{
    InitializeComponent();

    var dispatcher = Dispatcher;

    Auth.SetUserCredentials(...);
    _sampleStream = Stream.CreateSampleStream();
    _sampleStream.TweetReceived += (sender, args) =>
    {
        dispatcher.BeginInvoke(new Action(() =>
        {
            textBox.Text += args.Tweet.ToString();
        }));

    };
}

private void button_Click(object sender, RoutedEventArgs e)
{
    _sampleStream.StartStreamAsync();
}

private void button1_Click(object sender, RoutedEventArgs e)
{
    _sampleStream.StopStream();
}

[PsykotropyK]

As a matter of fact, the stream in my main app is stored as a private member of my class. So the Garbage Collector should not be the reason of my issue.

[linvi]

Does the code I sent to you work in WPF? This is how I would do it personally.

[PsykotropyK]

Linvi,

I adapted a bit your code to use it with a filtered stream (as I had no answer from the samplestream).

public partial class MainWindow : Window
    {
        private IFilteredStream _Stream;

        public MainWindow()
        {
            InitializeComponent();

            var dispatcher = Dispatcher;

            Auth.SetUserCredentials(...);
            _Stream = Stream.CreateFilteredStream();
            _Stream.AddTrack("#preorderpurpose");
            _Stream.AddTrack("#TheVoiceBrasil");
            _Stream.AddTrack("#AHSNoFX");
            _Stream.AddTrack("#aldubthirdmonthsary");
            _Stream.AddTrack("#buyconfidentalbumonitunes");

            _Stream.MatchingTweetReceived += (sender, args) =>
            {
                dispatcher.BeginInvoke(new Action(() =>
                {
                    textBox.Text += args.Tweet.ToString();
                }));

            };
        }

        private void button_Click(object sender, RoutedEventArgs e)
        {
            _Stream.StartStreamMatchingAllConditionsAsync();
        }

    }

This doesn't give me one hit, while using the same hashtags in my console tester gives me tens per seconds

[linvi]

Ok, the fact that even the SampleStream does not return anything implies something is really wrong with the connection.

Would you please verify that no error is being raised?

ExceptionHandler.SwallowWebExceptions = false;
var _Stream = Stream.CreateFilteredStream();
_Stream.StreamStarted += (sender, args) =>
{
    // Verify that this handler is called!
};

_Stream.StreamStopped += (sender, args) =>
{
    var exception = args.Exception;
    var disconnectMessage = args.DisconnectMessage;

    // Verify if you have any information in either of these variable
};

[PsykotropyK]

Hi Linvi,

I added your code in my app and there is indeed an exception:

Message = "Object reference not set to an instance of an object." Source = "Tweetinvi.WebLogic" StackTrace = "at Tweetinvi.WebLogic.TwitterClientHandler.SendAsync(ITwitterQuery twitterQuery, HttpRequestMessage request, CancellationToken cancellationToken) at Tweetinvi.WebLogic.TwitterClientHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpMessageInvoker.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken) at System.Net.Http.HttpClient.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption) at Tweetinvi.Streams.StreamTask.d__28.MoveNext()"

Let me know if you need more information. I will on my side start again looking from a blank WPF to an equivalent to my main app at which stage it start to bug (not sure when though I will have time though).

[PsykotropyK]

If it can help here is a larger code extract to show what I have:

From the View Model:

namespace TechnicalAnalyzer.Controls.Tools.Exchange
{
    public class TreeviewViewModel : DependencyObject
    {
private TwtCommand TwtCmd = new TwtCommand();

        private bool _streaming = false;
        public bool streaming 
        {
            get { return _streaming; } 
            set
            {
                if (value != _streaming)
                {
                    _streaming = value;
                    //TODO run or pause the stream
                    if (_streaming)
                    {
                        TwtCmd.StartStream();
                    }
                    else
                    {
                        TwtCmd.StopStream();
                    }
                }
            } 
        }
    }
}

In the TwtCommand (I didn't had time to test the dispatcher here yet)

namespace TechnicalAnalyzer.Data.Twitter
{
    class TwtCommand
    {
        private IFilteredStream MainStream;
        List<streamParameters> Criterias = new List<streamParameters>();
        int tester;

        public void StartStream()
        {
            if (MainStream == null)
            {
                MainStream = Stream.CreateFilteredStream();
            }
            var dbcmd = new DbCommand();
            Criterias = dbcmd.GetStreamingCriterias(); // This reads my database using EntityFramework to return the list of parameters
            RunStream(Criterias);
        }

        public void StopStream()
        {
            MainStream.StopStream();
            MainStream.ClearTracks();
        }

        private void RunStream(List<streamParameters> SearchCriterias)
        {

            if (SearchCriterias.Count > 0)
            {
                Auth.SetUserCredentials(...); 
                //TwitSetCredentials();
                for (int i = 0; i <= SearchCriterias.Count - 1; i++)
                {
                    MainStream.AddTrack(SearchCriterias[i].StreamCriteria);
                }

                MainStream.MatchingTweetReceived += (senderTwt, argsTwt) =>
                {
                    ReceivedTweet(argsTwt);
                };

                MainStream.StreamStarted += (senderTwt, argsTwt) =>
                {
                    tester = 1;
                };
                ExceptionHandler.SwallowWebExceptions = false;
                MainStream.StreamStopped += (senderTwt, argsTwt) =>
                {
                    var exception = argsTwt.Exception;
                    var disconnectMessage = argsTwt.DisconnectMessage;
                    tester = 2;
                };

                Thread StreamTask = new Thread(() => MainStream.StartStreamMatchingAllConditions());
                StreamTask.Start();
                StreamTask.IsBackground = true;
            }
        }

        private void ReceivedTweet(MatchedTweetReceivedEventArgs Tweet)
        {
            //DO STUFF
        }
    }
}

[linvi]

Wait a second here, are you using Tweetinvi version 0.9.10 or Tweetinvi 0.9.9.8 with this code?

[PsykotropyK]

0.9.10. But in order to be sure I just uninstall and reinstall it fully with NuGet ==> Same issue.

[PsykotropyK]

With the dispatcher code that you provided, and on my blank WPF, I have a different exception. Here is the code (i am using 0.9.10 too)

namespace TweetinviTester
{
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        private IFilteredStream _Stream;

        public MainWindow()
        {
            InitializeComponent();

            var dispatcher = Dispatcher;

            Auth.SetUserCredentials(...);
            _Stream = Stream.CreateFilteredStream();

            _Stream.AddTrack("#preorderpurpose");
            _Stream.AddTrack("#TheVoiceBrasil");
            _Stream.AddTrack("#AHSNoFX");
            _Stream.AddTrack("#aldubthirdmonthsary");
            _Stream.AddTrack("#buyconfidentalbumonitunes");

            _Stream.MatchingTweetReceived += (sender, args) =>
            {
                dispatcher.BeginInvoke(new Action(() =>
                {
                    textBox.Text += args.Tweet.ToString();
                }));

            };
            int tester;
            _Stream.StreamStarted += (senderTwt, argsTwt) =>
            {
                tester = 1;
            };
            ExceptionHandler.SwallowWebExceptions = false;
            _Stream.StreamStopped += (senderTwt, argsTwt) =>
            {
                var exception = argsTwt.Exception;
                var disconnectMessage = argsTwt.DisconnectMessage;
                tester = 2;
            };
        }

        private void button_Click(object sender, RoutedEventArgs e)
        {
            _Stream.StartStreamMatchingAllConditionsAsync();
        }

    }
}

Here is the exception (which is not immediately triggered, contrary to the one I post earlier on my main app):

Message = "A task was canceled." Source = "Microsoft.Threading.Tasks" argsTwt.Exception.StackTrace = " at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task) at Microsoft.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Microsoft.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() at Tweetinvi.Streams.StreamTask.d__28.MoveNext()"

[linvi]

I have just copied the exact same code and this is working properly for me. As a result would you please tell me which version of the .NET Framework you are using.

Also you told me that the stream is working properly in a console, are you testing the Console App from the same computer?

You can find my test example here:

https://github.com/linvi/tweetinvi.issues/releases/download/Version.0.9.10/WpfApplication1.zip

[linvi]

Is the solution working for you?

[PsykotropyK]

I will try to check in the afternoon.

[PsykotropyK]

Linvi,

The link gives a 404.

Just to be sure I started a new WPF from scratch and I still got the same error. I am using the .NET Framework 4.5, and the console is run from the same computer

I run the code from an other computer and got this error: Message = "Object reference not set to an instance of an object" Source = "Tweetinvi.Factories" StackTrace = " at Tweetinvi.Factories.Tweet.TweetFactory.GenerateTweetFromJson(String jsonTweet)\r\n at Tweetinvi.Streams.FilteredStream.b17_1(String json)\r\n at Tweetinvi.Streams.Helpers.StreamResultGenerator.<>c__DisplayClass20_0.b0(String json)\r\n at Tweetinvi.Streams.StreamTask.Start()"

[linvi]

I will have a look into this. By the way the link works properly for me. Please try again to download it form Github directly (not from email).

[PsykotropyK]

I manage to download it. I see that you use the .Net Framework 4.5.2, while I am using 4.5. I will update my framework and check.

[PsykotropyK]

I updated to 4.5.2 ==> same issues. I tried your file, with the 4.5.2 ==> same issues.

The exception remains different whether I am testing it on my laptop (A task was canceled) or on my desktop (object not set to an instance of an object).

For now I will adapt a console app to work in background and feed my database. Maybe I have something not properly installed or whatever on both of my computers. Don't quite get what it could be though. Unfortunately if you can't reproduce the exception, it may be hard to debug it :)

Thanks for the great support anyway, it's greatly appreciated. If you stop by Paris one day, send me a mp I''l fix you a beer or smthg.

[PsykotropyK]

I manage to fix the tester app. The good news is that it doesn't come from your code. The bad news is that I made you lost your time: I made a huge mistake copying the credentials

So now we will be able to focus again on the real issue, I will do as I said initially and reconstruct gradually an app similar to my main app and see when it starts to work unproperly. Until then, don't bother to check. And again sorry for the time lost. As I said I'll pay back in beers if you happen to come over to Paris,

[PsykotropyK]

Hi Linvi, So here is what I can say following my try to reproduce my main app (more or less) starting from the WPF app you provided. It seemed there is a lot of things that go wrong but I will ended up fixing everything like this.

1/ Exception can be raised on your WPF depending on the Tracks that I put on my stream: current top trends will generally trigger a “a task was cancelled” exception. Ex: if I use the Tracks as set initially in the WPF it works fine. If I put the first two or three keywords trends (as seen on http://trends24.in/) I trigger the exception, prior to receiving any tweets.

2/ I put the credentials in my App.config such as

<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.2" />
    </startup>
  <appSettings>
    <add key="ACCESS_TOKEN" value="…" />
    <add key="ACCESS_TOKEN_SECRET" value="…" />
    <add key="CONSUMER_KEY" value="…" />
    <add key="CONSUMER_SECRET" value="…" />
  </appSettings>
  <runtime>
…
  </runtime>
</configuration>

And access it using

Using System.Configuration
…..
public void SetCredentials()
        {
            Auth.SetUserCredentials(ConfigurationManager.AppSettings["CONSUMER_KEY"],
                    ConfigurationManager.AppSettings["CONSUMER_SECRET"],
                    ConfigurationManager.AppSettings["ACCESS_TOKEN"],
                    ConfigurationManager.AppSettings["ACCESS_TOKEN_SECRET"]);
        }

And adding a reference to System.Configuration.

I now again got an exception “The task got cancelled” using a list of tracks that were giving good results prior to changing my way of setting the credentials

It is quite hard to exactly understand what makes it bugs" it seemed to work fine again after I took out the reference to System.Configuration and modify my App.Config (especially by doing this last thing) but now again it raise an exception at every try while the original WPF you provided still works...

[linvi]

Hi again, tonight I had a user with a similar issue #64 as you can see above and we did a live conversation concerning the issue and it took around 5 minutes to identify what was wrong with the code.

Would you be up to do this?

[PsykotropyK]

linvi,

On my tester app, passing the credential still raise a task got cancelled exception if I try the top trend hashtag as sourced from http://trends24.in/

Here's my code

        public MainWindow()
        {

            InitializeComponent();

            var dispatcher = Dispatcher;

            _Stream = Stream.CreateFilteredStream();
            _Stream.Credentials = new TwitterCredentials(cred);
            _Stream.AddTrack("#itunes");

            _Stream.MatchingTweetReceived += (sender, args) =>
            {
                dispatcher.BeginInvoke(new Action(() =>
                {
                    textBox.Text += args.Tweet.ToString();
                }));

            };

            ExceptionHandler.SwallowWebExceptions = false;
            _Stream.StreamStopped += (senderTwt, argsTwt) =>
            {
                var exception = argsTwt.Exception;
                var disconnectMessage = argsTwt.DisconnectMessage;
            };
        }

        private void button_Click(object sender, RoutedEventArgs e)
        {
            _Stream.StartStreamMatchingAllConditionsAsync();
        }

Here, #itunes will work fine. On the other hand, using what I see on http://trends24.in/ (ex now, #DemiNoBrasil) will trigger the exception.

[PsykotropyK]

It also seem that the list of hashtag from my Main App is what triggers the exception. When I deploy this list on my tester app it raise the exception. Here's the list

            _Stream.AddTrack("#freemobile");
            _Stream.AddTrack("#freebox");
            _Stream.AddTrack("#total");
            _Stream.AddTrack("#cacib");
            _Stream.AddTrack("#creditagricole");
            _Stream.AddTrack("#amundi");
            _Stream.AddTrack("#lcl");
            _Stream.AddTrack("#peugeot");
            _Stream.AddTrack("#orange");
            _Stream.AddTrack("#citroen");
            _Stream.AddTrack("#ds");
            _Stream.AddTrack("#engie");
            _Stream.AddTrack("#vallourec");

I honestly have doubts that this list should trigger many hits per seconds.

[linvi]

I will try this tonight. The issue only occurs on WPF project? Also is it happening with the project I gave to you?

[PsykotropyK]

It doesn't seem to bug from your code... kind of weird regarding that both are using the same sets of references, similar app.config and app.xaml, with same kind of code and same .NET frtamework

[PsykotropyK]

So i started again from your code, which works fine, and edited the App.config to add some keys to later use them to recover the credentials :

<appSettings>
    <add key="ACCESS_TOKEN" value="..." />
    <add key="ACCESS_TOKEN_SECRET" value="..." />
    <add key="CONSUMER_KEY" value="..." />
    <add key="CONSUMER_SECRET" value="..." />
  </appSettings>

At this stage, and while I didn't yet made any changes to the code behind (except to put the trending hashtags) the code crash with the a task got cancelled exception.

If revert the app.config to its original state it starts working fine again.

[PsykotropyK]

Still from a working version of your code, if I add a reference to the "System.Configuration" library, it crashes too (and this without any other changes to your code, except the hashtags list, nor yet any changes in the App.Config).

If I revert the change (delete the reference) the app still crashes. Is there any chances that a file kept a reference to the System.Configuration library? If so how to fix? I will anyway try again in the evening just to clear the risk to have been blacklisted by Twitter for too many stream open and close attempt (even though I didn't exceeded 6)

I guess this is what makes my main app and the tester app I was using yesterday to crash (both have or used to have a reference to the System.Configuration library)

[linvi]

Sorry, I have been quite busy yesterday and I might be today too but I will do my best to test this.

[linvi]

Hi there, so I have tested with the reference to the System.Configuration without being able to reproduce your issue.

Do you think you'd be able to share your project?

[PsykotropyK]

You can download the project here: .7z(~3mo): http://dl.free.fr/getfile.pl?file=/ycR7IeQT .zip(~10mo): http://dl.free.fr/sDtdPHmwx

The page is in French, clic on "Valider et telecharger le fichier" (=Validate and download the file) I just tested it now and it was not working.

As it seems to bug only when the set of Tracks trigger a lot of hits, be sure to replace the one in the project by the 3/4 first "trends" from http://trends24.in/

[PsykotropyK]

Linvi, did you manage to reproduce the issue using the project I shared? I didn't work much on this project lately, and anyway I will try to put it in a dedicated project in order to avoid any bad interactions with other reference I may use in my main app, still I am interested in understanding what is going wrong. Thanks

[linvi]

Hi, I am sure you have noticed I am quite slow to reply, as I currently have very little time. I have not received any issue during my last tests but I will try again before I release a new version.

I will keep you up to date.

[risingsudhir]

I am also getting same problem - a simple console application, trying to use filter stream, going through the same cycle of events as mentioned above.

I added event handler for json object received and from the message, it appears twitter is denying the request.

e.Json =

any clue what could be possibly wrong? I am using same authentication tokens in R and I can get the data from twitter in R.

[PsykotropyK]

@risingsudhir Are you sure you have the same issue as me? My problems come from the use of a reference to System.Configuration. I have no problem using a console app and solve my issue in my main app by transfering all the Twitter commands into a dedicated project (which only reference what's needed for Tweetinvi).

[risingsudhir]

sorry, I mean the sequence of events. I am not referencing System.Configuration anywhere, neither using WPF application. Tokens are hard coded in the application.

Did you try binding an handler for JsonObjectReceived event?

[PsykotropyK]

For what it looks like, we are not talking of the same issue: everytging works perfectly on a console app for me. I think the best would be for you to create a dedicated topic, and post your code. Rgds

[linvi]

Hi there. I just wanted to let you know that the issue has potentially been fixed in the coming version 0.9.10.1. I will keep you up to date when it is released.

[linvi]

To verify in next release

[PsykotropyK]

Sure let me know and thanks + happy new year

[linvi]

Hello there, I have tried your application without even changing the DLL version and everything is working as expected.

[linvi]

Please verify that you still encounter this issue with the latest version of Tweetinvi. In the meantime I am closing this thread.