search

StackStorm-Exchange / stackstorm-google

st2 content pack containing google integrations
https://exchange.stackstorm.org/
Apache License 2.0
3 stars 4 forks source link

AttributeError: 'NoneType' object has no attribute 'text' #2

Open timsk-ru opened 7 years ago

timsk-ru commented 7 years ago

Receiving the above error on running from Stackstorm 2.3.1 via any method. Full error text:

result :
--------
result : None
stderr :
  Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py", line 259, in <module>
    obj.run()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py", line 155, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/google/actions/get_search_results.py", line 18, in run
    results = google.search(query, num_page)[:count]
  File "/opt/stackstorm/virtualenvs/google/src/google-search-api/google/modules/standard_search.py", line 69, in search
    res.name = _get_name(li)
  File "/opt/stackstorm/virtualenvs/google/src/google-search-api/google/modules/standard_search.py", line 86, in _get_name
    return a.text.strip()
AttributeError: 'NoneType' object has no attribute 'text'
exit_code : 1
LindsayHill commented 7 years ago

Thanks for the report. Might be something changed again on the Google side. Will take a look

LindsayHill commented 7 years ago

I just ran this set of commands with StackStorm 2.3.2, and it seems to be working:

vagrant@bwcvagrant:~$ st2 --version
st2 2.3.2
vagrant@bwcvagrant:~$ st2 pack list |grep goo
vagrant@bwcvagrant:~$ st2 pack install google

    [ succeeded ] download pack
    [ succeeded ] make a prerun
    [ succeeded ] install pack dependencies
    [ succeeded ] register pack

+-------------+-------------------------------------------------+
| Property    | Value                                           |
+-------------+-------------------------------------------------+
| name        | google                                          |
| description | st2 content pack containing google integrations |
| version     | 0.2.0                                           |
| author      | StackStorm, Inc.                                |
+-------------+-------------------------------------------------+
vagrant@bwcvagrant:~$ st2 run google.get_search_results query=lkhill
..
id: 597ec881bb0f18080d538006
status: succeeded
parameters:
  query: lkhill
result:
  exit_code: 0
  result: '[{"thumb": null, "google_link": "http://www.google.com/url?q=https://lkhill.com/&sa=U&ved=0ahUKEwiF39T47LLVAhUE2LwKHa9XB04QFggUMAA&usg=AFQjCNE9KoFzODwGyi_2G1uzoa7LWEJR1A", "link": null, "name": "Home \u00b7 Lindsay Hill", "description": "Recruiters: Must Try Harder. 01 Jul 2017. Right now, it''s an employee''s market in \nthe Bay Area. Technology firms are growing, and they''re always trying to hire\u00a0..."}, {"thumb": null, "google_link": "http://www.google.com/url?q=https://www.facebook.com/lkhillbooks/&sa=U&ved=0ahUKEwiF39T47LLVAhUE2LwKHa9XB04QFggZMAE&usg=AFQjCNEfSPgrL0tN5Mz0Vu-V7uRrNxtl-g", "link": null, "name": "L.K. Hill - Home | Facebook", "description": "L.K. Hill. 561 likes. Author L.K.Hill writes historical fiction, scifi/fantasy, and crime \ndrama. Be sure to check out the pages for her Street Games and..."}, {"thumb": null, "google_link": "http://www.google.com/url?q=http://lkhill.blogspot.com/&sa=U&ved=0ahUKEwiF39T47LLVAhUE2LwKHa9XB04QFggfMAI&usg=AFQjCNF4DZsyiA_5bn4v-HEaObkt-cvLZA", "link": null, "name": "L.K.Hill", "description": "21 Sep 2015 ... I''ve decided to streamline my two blogs, this one (LKHill) and the second one (\nMusings on Fantasia). I''m trying to figure out a custom redirect,\u00a0..."}, {"thumb": null, "google_link": "http://www.google.com/url?q=https://www.goodreads.com/author/show/7234522.L_K_Hill&sa=U&ved=0ahUKEwiF39T47LLVAhUE2LwKHa9XB04QFggmMAM&usg=AFQjCNEcZsBMYulhddELwmLcU8S0r3U_8A", "link": null, "name": "L.K. Hill (Author of The Botanist) - Goodreads", "description": "Goodreads Author.\n ... Her crime and historical fiction are written under her initials, L.K., while her scifi/fantasy and dystopian are written under her full name, Liesel K. Hill.\n ... Hill hasn''t written any blog posts yet."}, {"thumb": null, "google_link": "http://www.google.com/url?q=http://musingsonfantasia.blogspot.com/&sa=U&ved=0ahUKEwiF39T47LLVAhUE2LwKHa9XB04QFggrMAQ&usg=AFQjCNHViHNUL-1TWtiPuBic33wVyCiK9w", "link": null, "name": "L.K.Hill''s Musings on Fantasia", "description": "Facebook Twitter Google+ Pinterest Share. Posted by L.K.Hill, Author at 9:28 AM \n4 comments : Labels: Desolate Mantle Contest , Dragon Magic , Holiday Stuff\u00a0..."}, {"thumb": null, "google_link": "http://www.google.com/url?q=https://lkhillbooks.tumblr.com/&sa=U&ved=0ahUKEwiF39T47LLVAhUE2LwKHa9XB04QFggwMAU&usg=AFQjCNHHfJ_7j0QpgQOzfQRWuLMOKTgVFA", "link": null, "name": "Author L.K.Hill", "description": "L.K. Hill is an author who writes across multiple genres, including scifi/fantasy, \nhistorical fiction, and crime fiction. She writes under her initials (historical and\u00a0..."}, {"thumb": null, "google_link": "http://www.google.com/url?q=https://www.amazon.com/L.K.-Hill/e/B00B2BSC9C&sa=U&ved=0ahUKEwiF39T47LLVAhUE2LwKHa9XB04QFgg1MAY&usg=AFQjCNF6pd--edZiWRHfycP4qKCwqKremA", "link": null, "name": "Amazon.com: L.K. Hill: Books, Biography, Blog, Audiobooks, Kindle", "description": "L.K. Hill (1964 to present) is a long time resident of Ct. and is a certified nurse''s \naide. I''ve always had an interest in the Vikings, and wanted to put to rest some of\n\u00a0..."}, {"thumb": null, "google_link": "http://www.google.com/url?q=https://twitter.com/lkhillbooks%3Flang%3Den&sa=U&ved=0ahUKEwiF39T47LLVAhUE2LwKHa9XB04QFgg6MAc&usg=AFQjCNEBAq6Ti47EK9m2WQGngb-hRk8ABg", "link": null, "name": "L K Hill (@lkhillbooks) | Twitter", "description": "The latest Tweets from L K Hill (@lkhillbooks). Passionate writer of dystopian \nfantasy, historical fiction, and crime drama. I create characters you can''t get \nenough\u00a0..."}, {"thumb": null, "google_link": "http://www.google.com/url?q=http://www.allmusic.com/artist/lk-hill-mn0000122590&sa=U&ved=0ahUKEwiF39T47LLVAhUE2LwKHa9XB04QFghAMAg&usg=AFQjCNHnsKRDF2gRnHH1RkVlCKoEnZoH7A", "link": null, "name": "L.K. Hill | Credits | AllMusic", "description": "Find L.K. Hill credit information on AllMusic. ... L.K. Hill. Credits \u2193. Share this \npage. facebook \u00b7 twitter \u00b7 google+. Credits. (1-1 of 1). Year, Album, Artist. All \nCredits\u00a0..."}, {"thumb": null, "google_link": "http://www.google.com/url?q=https://www.youtube.com/watch%3Fv%3DR7P-QeoVk5s&sa=U&ved=0ahUKEwiF39T47LLVAhUE2LwKHa9XB04QtwIIRjAJ&usg=AFQjCNF4dVS4MVBs-irpUH18GY9XqI2D1w", "link": null, "name": "Frey''s Saga by L.K. Hill - 60 sec - YouTube", "description": null}]'
  stderr: ''
  stdout: ''
vagrant@bwcvagrant:~$

Could you do a quick check with 2.3.2, and see if it's still an issue? Otherwise we'll dig deeper into what's not working on your system

(FYI, my system is ST2 2.3.2 on Ubuntu 14.04)

LindsayHill commented 7 years ago

Also: Is there any else 'special' about your environment? E.g. proxy servers?

Kami commented 7 years ago

It's also worth noting that this action depends on parsing raw HTML of google.com since they don't really provide an API which makes it prone to breaking in case Google changes the markup.

Kami commented 7 years ago

It might also be something specific to the search string / keyword you are using so would be a good idea to also include that here.

timsk-ru commented 7 years ago

Hi - thanks for looking at this so quickly. The environment is 2.3.1, only installed a few days ago (just prior to the 2.3.2 release) into a clean standalone Ubuntu 14.04 instance via the one line script install method. The only thing 'special' about the install has been getting Hubot to use a Discord Adapter (works like a treat by the way) as well as swapping out the default self-signed certs for some Let's Encrypt ones post install to avoid browser nags. Everything else seems to work ok. I can snapshot this and update to 2.3.2 if you'd still like me to.

However, just picked up on something to feedback. Just for completeness I checked with the st2 run Lindsay posted above and it returned expected - i.e. fully correct - results. Re-running this with shorter query parameters generates the original error. This seems to occur at <=4 characters, so a search for 'cat' or 'doge' will both fail, but a search for 'timsk' won't. query="doge much funny" is of course fine ;-)

So, something less than originally thought, but my first port of call with testing anything is always 'cat', hence why picking this up straight away.

I'm most interested in pulling apart the imported google scripts for similar web-scraping purposes - would just love it if there was a simple addition to the http-request runner to return say a json array of the html tree to pick out results from.

Hope this helps (and feel free to poke fun for my terribly unimaginative search habits).

LindsayHill commented 7 years ago

Just did a little more testing - it's not even as simple as <= 4 chars

Search term 'dog' works, but not 'cat'. But 'catty' also fails. 'cats dogs' fails, 'cats dogs living together' works.

Must be something else going on here, other than just search term length.

LindsayHill commented 7 years ago

Some more digging - it is a problem with pages that show a definition of the term at the top, or ones that have a "See results about" box on the right - as 'cat' does, when it offers searches for Caterpillar.

Looks like maybe the library we're using here needs an update. It's a bit of problem with the way our web scraping is done. Needs to use an 'official' library, if there is one.

timsk-ru commented 7 years ago

Ah, yes I see how that would affect things. Looks like there are some official libraries available, though needing a sign up for Custom Search (free with limits) https://github.com/google/google-api-python-client and https://developers.google.com/custom-search/json-api/v1/libraries

swastikpanigrahi7 commented 5 years ago

I am running st2 2.6.0. and getting empty output when i run the st2 run cmd for google pack.

root@stackstorm-bf44b7f9f-26ms2:/opt/stackstorm/packs/google# st2 run google.get_search_results "query=lkhill" . id: 5d48fdb5ff23af0124e8977c status: succeeded parameters: query: lkhill result: exit_code: 0 result: '[]' stderr: '' stdout: '' root@stackstorm-bf44b7f9f-26ms2:/opt/stackstorm/packs/google# st2 --version st2 2.6.0 root@stackstorm-bf44b7f9f-26ms2:/opt/stackstorm/packs/google#

can someone help on this issue? I have a VPN connected while working on this.