add tweet count logger to stream.py #14

[Coder-Manan]

A similar PR may already be submitted! Please search among the Pull request before creating one.

Thanks for submitting a pull request! Please provide enough information so that others can review your pull request:

For more information, see the CONTRIBUTING guide.

• What kind of change does this PR introduce? (Bug fix, feature, docs update, ...) feature

• What is the current behavior? (You can also link to an open issue here)

  14

• What is the new behavior (if this is a feature change)? Log no of tweets written to tweets.json after every thousand writes

• Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

• Other information:

Explain the motivation for making this change. What existing problem does the pull request solve?

Test plan (required) No plan Demonstrate the code is solid. Example: The exact commands you ran and their output, screenshots / videos if the pull request changes UI.

if (self.counter % 1000 == 0):
            print("No of tweets currently in tweets.json = ", self.counter)

Code formatting

Closing issues

Fixes #14

[Coder-Manan]

Currently I am printing the number of tweets after every thousand writes

[kaustav202]

Also add logging on smaller count like 50.. it takes a long time to reach 1000 tweets

[Coder-Manan]

Also add logging on smaller count like 50.. it takes a long time to reach 1000 tweets

Fixed it

[Coder-Manan]

I am getting the following error when trying to run the app

Traceback (most recent call last):
  File "C:\Python310\lib\site-packages\pandas\core\indexes\base.py", line 3800, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas\_libs\index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'text'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\xxxx\xxxx\RealTime-TwitterDataAnalysis\app\main.py", line 29, in <module>
    python = check_word_in_tweet('#google', ds_tweets)
  File "D:\xxxx\xxxx\RealTime-TwitterDataAnalysis\app\main.py", line 16, in check_word_in_tweet
    contains_column = data['text'].str.contains(word, case = False)
  File "C:\Python310\lib\site-packages\pandas\core\frame.py", line 3805, in __getitem__
    indexer = self.columns.get_loc(key)
  File "C:\Python310\lib\site-packages\pandas\core\indexes\base.py", line 3802, in get_loc
    raise KeyError(key) from err
KeyError: 'text'

[kaustav202]

I am getting the following error when trying to run the app

Traceback (most recent call last):
  File "C:\Python310\lib\site-packages\pandas\core\indexes\base.py", line 3800, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas\_libs\index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'text'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\xxxx\xxxx\RealTime-TwitterDataAnalysis\app\main.py", line 29, in <module>
    python = check_word_in_tweet('#google', ds_tweets)
  File "D:\xxxx\xxxx\RealTime-TwitterDataAnalysis\app\main.py", line 16, in check_word_in_tweet
    contains_column = data['text'].str.contains(word, case = False)
  File "C:\Python310\lib\site-packages\pandas\core\frame.py", line 3805, in __getitem__
    indexer = self.columns.get_loc(key)
  File "C:\Python310\lib\site-packages\pandas\core\indexes\base.py", line 3802, in get_loc
    raise KeyError(key) from err
KeyError: 'text'

Check your tweets.json file.. most likely it has no data. you need to run stream.py and then run main after a gap

[kaustav202]

Also fetch the latest from upstream before proceeding

[Coder-Manan]

Ok. I will look into it today

[Coder-Manan]

Please check if logging is happening now or not, I have changed the location where I was printing the counter

[Coder-Manan]

@kaustav202