Error in open.connection(x, "rb")

[aureincm]

Hi there,

first of all, thank you so much for this library!! I discovered it recently and it really made my month!!!! :-)

I am opening this issue because I am encountering a connection error when trying to run ngram() within a loop over a large number of ngrams.

I have more than 600 Ngrams to go through, and a 'for' loop would work perfectly for me, except, that the connection to the google.book webpage seems to be lost after about 75 Ngrams.

I thought you may have a hint for me to try and fix this issue...

Any clue would be greatly appreciated! Thanks again!

• Aurelie

Here is the error message I get

Error in open.connection(x, "rb") : 
  cannot open the connection to 'https://books.google.com/ngrams/graph?content=lenteur+s%27oppose+%C3%A0+la+rapidit%C3%A9&corpus=30&year_start=1800&year_end=2019&smoothing=0&case_insensitive=on'

Here is some reproducible code:

Ngram_data <- structure(list(id_overall = c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 
11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 
27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 
43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 
59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 
75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 
91, 92, 93, 94, 95, 96, 97, 98, 99, 100), Xgram_pattern = c("écolier marche sur la banquise", 
"métal sert à faire des bouchons", "canaille utilise sa carapace", 
"chiffre se multiplie par la largeur", "dicton est un proverbe", 
"éléphant utilise sa trompe", "entaille est une coupure", "estomac permet la digestion", 
"seau permet un bon arrosage", "châle est mis pour le carnaval", 
"mouches envahissent ce camping", "gratin est garni de jambon", 
"entretien donne suite à une embauche", "formule est utilisée en mathématiques", 
"cigale demande à la fourmi", "habitudes entrainent une routine", 
"blason est un aigle", "main remplace la rame", "terreau se mélange au thym", 
"papyrus est une copie", "batterie remplace les piles", "beurre est du gras", 
"biche donne naissance à un faon", "bougie s'écoule de la cire", 
"disgrâce n'est pas une qualité", "matériel mesure la vitesse", 
"compère emmène sa maîtresse", "benne ramasse les ordures", 
"gruyère est rangé avec les fromages", "brevet est obtenu à la fin du collège", 
"tour est construite par une araignée", "torchons sont rangés avec les serviettes", 
"enceinte regorge de lions", "garçon indique la note", "matière est utilisée avec la règle", 
"sourd est également muet", "soutien est une aide", "tribunal est le lieu de travail des juges", 
"caoutchouc sert à fermer le pot", "viande est du boeuf", "préposés ordonnent aux enfants", 
"tache provient du soleil", "évènement a été provoqué par une voiture", 
"fichier est constitué de plusieurs pièces", "cendrier contient des cigarettes", 
"cernes traduisent un manque de sommeil", "nid a été construit par un oiseau", 
"tarot se pratique avec des cartes", "voisine lit mon avenir", 
"album contient des photos", "coffret a été ouvert à la mort", 
"dessin représente son amour", "minuteur sonne le matin", "voisin n'est pas un ami", 
"songe est un rêve", "plombage est posé sur la dent", "épouse demande à son mari", 
"oncle se promène avec ma tante", "virtuoses éclairent la nuit", 
"bague entoure le doigt", "cageot regorge de raisin", "mouvement se fait dans la piscine", 
"bête mange des carottes", "vermine a mangé la noisette", "flageolet est un haricot", 
"framboise ressemble à une fraise", "gigot provient d'un agneau", 
"goéland ressemble à une mouette", "loisir se joue avec des baguettes", 
"chimère utilise ses tentacules", "village est construit par des esquimaux", 
"barque permet le sauvetage", "informaticien répare des ordinateurs", 
"innocent s'oppose à un coupable", "hamac permet de faire une sieste", 
"lenteur s'oppose à la rapidité", "personne est une star", 
"insecte est un ver", "homme utilise la magie", "planche permet une nage", 
"caleçon remplace le slip", "canards se baignent dans la mare", 
"challenge est un défi", "chouette diffère du hibou", "truc est un couteau", 
"plaintes témoignent de sa tristesse", "politiciens marquent les limites", 
"patron indique aux employés", "bombe entraine une explosion", 
"arc est utilisé avec des flèches", "congrès précède le vendredi", 
"firme est une entreprise", "bracelet est fabriqué avec du sucre", 
"place accueille les fous", "salon est un bar", "tétine est saisie par le bébé", 
"rosier produit des roses", "beau retrouve sa belle", "terrain est un parc", 
"carré diffère d'un rond"), Xgram_dependencies = c("banquise=>écolier", 
"bouchons=>métal", "carapace=>canaille", "largeur=>chiffre", 
"proverbe=>dicton", "trompe=>éléphant", "coupure=>entaille", 
"digestion=>estomac", "arrosage=>seau", "carnaval=>châle", "camping=>mouches", 
"jambon=>gratin", "embauche=>entretien", "mathématiques=>formule", 
"fourmi=>cigale", "routine=>habitudes", "aigle=>blason", "rame=>main", 
"thym=>terreau", "copie=>papyrus", "piles=>batterie", "gras=>beurre", 
"faon=>biche", "cire=>bougie", "qualité=>disgrâce", "vitesse=>matériel", 
"maîtresse=>compère", "ordures=>benne", "fromages=>gruyère", 
"collège=>brevet", "araignée=>tour", "serviettes=>torchons", 
"lions=>enceinte", "note=>garçon", "règle=>matière", "muet=>sourd", 
"aide=>soutien", "juges=>tribunal", "pot=>caoutchouc", "boeuf=>viande", 
"enfants=>préposés", "soleil=>tache", "voiture=>évènement", 
"pièces=>fichier", "cigarettes=>cendrier", "sommeil=>cernes", 
"oiseau=>nid", "cartes=>tarot", "avenir=>voisine", "photos=>album", 
"mort=>coffret", "amour=>dessin", "matin=>minuteur", "ami=>voisin", 
"rêve=>songe", "dent=>plombage", "mari=>épouse", "tante=>oncle", 
"nuit=>virtuoses", "doigt=>bague", "raisin=>cageot", "piscine=>mouvement", 
"carottes=>bête", "noisette=>vermine", "haricot=>flageolet", 
"fraise=>framboise", "agneau=>gigot", "mouette=>goéland", "baguettes=>loisir", 
"tentacules=>chimère", "esquimaux=>village", "sauvetage=>barque", 
"ordinateurs=>informaticien", "coupable=>innocent", "sieste=>hamac", 
"rapidité=>lenteur", "star=>personne", "ver=>insecte", "magie=>homme", 
"nage=>planche", "slip=>caleçon", "mare=>canards", "défi=>challenge", 
"hibou=>chouette", "couteau=>truc", "tristesse=>plaintes", "limites=>politiciens", 
"employés=>patron", "explosion=>bombe", "flèches=>arc", "vendredi=>congrès", 
"entreprise=>firme", "sucre=>bracelet", "fous=>place", "bar=>salon", 
"bébé=>tétine", "roses=>rosier", "belle=>beau", "parc=>terrain", 
"rond=>carré")), row.names = c(NA, -100L), class = c("tbl_df", 
"tbl", "data.frame"))

# extract and store Google Ngram frequencies for each Ngram_pattern
for (i in 1:nrow(Ngram_data)) 
  {
  # launch Google Ngram query and extract the results that are displayed with viewer
  google_ngram_tmp <- ngram(Ngram_data$Xgram_pattern[i],  
                            corpus = "fre_2019",
                            year_start = 1800,
                            year_end = 2019,
                            smoothing = 0,
                            case_ins = TRUE, 
                            aggregate = TRUE)

  closeAllConnections()

  # returns the Ngram index at each iteration to keep track of the computation
  print (paste("Ngram index", i, sep = ": ") )
  # print(showConnections(all = FALSE))
}

[seancarmody]

Hi Aurelie

I think that the problem you are running into is that the Google rate limits the requests to the ngram viewer: if you hit it with too many requests it will block your IP address's access for a period of time (if you visit https://books.google.com/ngrams after your error you'll find it says "Please try again later").

I will look into providing a more informative error message, but the most constructive suggestion I would have is to break your work into chunks. Since you managed to get through to 75, you could try looping 1:50, saving the results and then 51:100 and combining what you get.

Alternatively, you can download the full ngram sets here: https://storage.googleapis.com/books/ngrams/books/datasetsv3.html. However, you'll have to work through them yourself as I haven't written any code to help with that!

Sean.

[seancarmody]

Aurelie,

I have had a closer look and here is a quick and dirty fix, adding some periods of "sleep" to avoid hitting the rate limit:

google_ngram_tmp <- NULL

for (ixs in chunk(1:nrow(Ngram_data), 50)){
  for (i in ixs) 
  {
    # launch Google Ngram query and extract the results that are displayed with viewer
    google_ngram_tmp <- bind_rows(google_ngram_tmp, 
                                  ngram(Ngram_data$Xgram_pattern[i], 
                                      corpus = "fre_2019", 
                                      year_start = 1800,
                                      year_end = 2019,
                                      smoothing = 0,
                                      case_ins = TRUE,
                                      aggregate = TRUE)
    )

    # returns the Ngram index at each iteration to keep track of the computation
    print (paste("Ngram index", i, sep = ": ") )
  }
  print("Sleeping for a while...")
  Sys.sleep(120)
}

class(google_ngram_tmp) <- c("ngram", class(google_ngram_tmp))

This could be improved (e.g. there's no need to sleep after the last loop), but it does the job.

Separately I noticed that, despite all of this hard work, most of these phrases will give no results (I only found results for four of the phrases). It's worth noting that there are no ngrams in Google's collection that are more than 5 words long, and some of these phrases may simply not appear. Did you mean to use Xgram_dependencies instead?

[aureincm]

Hi Sean,

thank you very much for your fast reply.

This was really helpful, because I was looking towards an R problem, and did not even think about a webpage rate limit.

As you suggested, I added a sleeping period at the end of each loop, and it works perfectly (I had to add a longer period of 600sec though) 👍

To answer your question, yes, I am aware that most of the Xgram_pattern entries are either too long or do not appear in the google database. This is actually part of my research question, which aims at comparing pattern vs. dependencies. So of course, I also plan to run a similar piece of code with Xgram_dependencies, which will be as easy as pie thanks to your code :-)

BTW, I should also mention that the using your library allows me to run some fancy analysis I gave up on because I did not have enough computer storage to load the entire ngram dataset.... Thanks again for this nice piece of code!!

Cheers

• Aurelie

[seancarmody]

Happy to help Aurelie - I'm glad the package is proving useful.