ropensci / rdataretriever

R interface to the Data Retriever
https://docs.ropensci.org/rdataretriever
Other
44 stars 21 forks source link

"Failed to load script: gentry" error regardless of dataset I try to load #66

Closed ahhurlbert closed 8 years ago

ahhurlbert commented 8 years ago

I just reinstalled retriever, and reinstalled the ecoretriever R package. Here is what I get when I try to fetch any data package:

> install.packages('ecoretriever')
Installing package into ‘C:/Users/hurlbert/Documents/R/win-library/3.2’
(as ‘lib’ is unspecified)
trying URL 'http://cran.rstudio.com/bin/windows/contrib/3.2/ecoretriever_0.2.1.zip'
Content type 'application/zip' length 21716 bytes (21 KB)
downloaded 21 KB

package ‘ecoretriever’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
    C:\Users\hurlbert\AppData\Local\Temp\RtmpWWnosc\downloaded_packages
> library(ecoretriever)

  Use get_updates() to download the most recent release of download scripts

    New to ecoretriever? Examples at
      https://github.com/ropensci/ecoretriever/
      Use citation(package='ecoretriever') for the package citation

Use suppressPackageStartupMessages() to suppress these messages in the future
Warning message:
package ‘ecoretriever’ was built under R version 3.2.3 
> get_updates()
Please wait while the retriever updates its scripts, ...
The retriever scripts are up-to-date with the most recent official release!
Downloaded scripts: AmnioteLH.script, baad.py, Baldridge2012.script, Dryad_tamburello_etal_2013.script, 
EA_anderson2011.script, EA_Chuetal2013.script, EA_Colin2014.script, EA_Fray_Jorge2013.script, EA_Luo.script, 
EA_lynch2013.script, EA_Matter_2014.script, EA_palmer2007.script, EA_Pelissier2011.script, 
EA_Rodriguez-Buritica2013.script, EA_strona2013.script, EA_Turcotte.script, EA_Wilman_2014.script, 
eBird_observation.py, gwdd.py, prism_4km_monthly_modern.py, vertnet.py
> ecoretriever::datasets()
 [1] "Available datasets : 45"                                                        
 [2] "Adler2007"                                                                      
 [3] "AmnioteDB"                                                                      
 [4] "Anderson2011"                                                                   
 [5] "ArcticBreedBird"                                                                
 [6] "AvianBodySize"                                                                  
 [7] "BAAD"                                                                           
 [8] "BBS"                                                                            
 [9] "BBS50"                                                                          
[10] "Bioclim"                                                                        
[11] "CForBioData"                                                                    
[12] "Clark2006"                                                                      
[13] "DelMoral2010"                                                                   
[14] "eBirdOD"                                                                        
[15] "EltonTraits"                                                                    
[16] "FIA"                                                                            
[17] "FishParasiteHosts"                                                              
[18] "FrayJorge"                                                                      
[19] "GWDD"                                                                           
[20] "HomeRanges"                                                                     
[21] "leaf_herbivory"                                                                 
[22] "MammalDIET"                                                                     
[23] "MammalLH"                                                                       
[24] "MammalMR2010"                                                                   
[25] "mammsupertree"                                                                  
[26] "MarineSize"                                                                     
[27] "Matter2014"                                                                     
[28] "MCDB"                                                                           
[29] "McGlinn2010"                                                                    
[30] "MiscAbundanceDB"                                                                
[31] "MoM2003"                                                                        
[32] "NPN"                                                                            
[33] "Palmer2007"                                                                     
[34] "Pantheria"                                                                      
[35] "Petraitis2008"                                                                  
[36] "PhytoplankonBiovolume"                                                          
[37] "PlantTaxonomy"                                                                  
[38] "PortalMammals"                                                                  
[39] "PRISM"                                                                          
[40] "Ramesh2010"                                                                     
[41] "SonoranPerennials"                                                              
[42] "Steppe_plants_2013"                                                             
[43] "TreeWesternGhats"                                                               
[44] "vertnet"                                                                        
[45] "Woods2009"                                                                      
[46] "Zachmann2010"                                                                   
[47] "Groups:"                                                                        
[48] "ALL"                                                                            
[49] "ALLOCATION"                                                                     
[50] "ANIMALS"                                                                        
[51] "BIOMASS"                                                                        
[52] "BIOME"                                                                          
[53] "BIRDS"                                                                          
[54] "BUTTERFLIES"                                                                    
[55] "CHINA"                                                                          
[56] "CLIMATE"                                                                        
[57] "COMPILATION"                                                                    
[58] "CONTINENTAL"                                                                    
[59] "DATA TYPE"                                                                      
[60] "DESERT"                                                                         
[61] "EXPERIMENTAL"                                                                   
[62] "FISH"                                                                           
[63] "FOREST TYPE"                                                                    
[64] "GLOBAL"                                                                         
[65] "LOCAL"                                                                          
[66] "MAMMALS"                                                                        
[67] "METAPOPULATION"                                                                 
[68] "NECROMASS"                                                                      
[69] "OBSERVATION"                                                                    
[70] "OBSERVATIONAL"                                                                  
[71] "PHENOLOGY"                                                                      
[72] "PHYTOPLANKTON"                                                                  
[73] "PLANTS"                                                                         
[74] "REGIONAL"                                                                       
[75] "SPATIAL SCALE"                                                                  
[76] "STAND STRUCTURE"                                                                
[77] "TAXON"                                                                          
[78] "TAXONOMY"                                                                       
[79] "TIME SERIES"                                                                    
[80] "Failed to load script: gentry (C:/Users/hurlbert/Documents/.retriever/scripts/)"
> mcg = ecoretriever::fetch("McGlinn2010")
Failed to load script: gentry (C:/Users/hurlbert/Documents/.retriever/scripts/)
> mcdb = ecoretriever::fetch("MCDB")
Failed to load script: gentry (C:/Users/hurlbert/Documents/.retriever/scripts/)

However, I WAS successfully able to install a dataset from the command line.

Note that my student gets basically the same message at the command line when typing retriever --version:

Failed to load script: gentry (C:/Users/hurlbert/Documents/.retriever/scripts/)v1.8.3

ethanwhite commented 8 years ago

Looks like maybe you both have an old cached version of the gentry script. Try running

retriever reset scripts

from the command line and then updating the scripts using either

retriever update

from the command line or

get_updates()

from R.

henrykironde commented 8 years ago

was wondering if that has anything to do with the Cedral site BLOHMR site data

ethanwhite commented 8 years ago

was wondering if that has anything to do with the Cedral site BLOHMR site data

I suspect it's just residual issues from https://github.com/weecology/retriever/issues/402, but we'll know if it's something trickier once they try the reset.

ahhurlbert commented 8 years ago

Ok, I'm on a different outdated machine at the moment. In R I tried ecoretriever::fetch('BBS50') and it downloaded only the species and routes files. So clearly an old install (I think 1.7.0)

In the command line, I reinstalled using pip, then retriever reset scripts and retriever update. Things seemed to go ok, and I can see the scripts file has all newly downloaded scripts in it.

But in R, I get the same problem as before (and it looks like get_updates() isn't quite working properly):

> get_updates()
Please wait while the retriever updates its scripts, ...
The retriever scripts are up-to-date with the most recent official release!
Error in sort.int(x, na.last = na.last, decreasing = decreasing, ...) : 
  'x' must be atomic
> mcg = ecoretriever::fetch('McGlinn2010')
Failed to load script: gentry (C:/Users/hurlbert/Documents/.retriever/scripts/)

I'll note that I closed out of both RStudio and Git Bash and then reopened them before trying this.

ethanwhite commented 8 years ago

OK, thanks.

Can you paste the output of retriever --version here and attach the ~/.retriever/scripts/gentry.py file?

ahhurlbert commented 8 years ago

Ok, back on my original machine--same result as above.

$ retriever --version
v1.8.3

gentry.py:

#retriever
"""Retriever script for Alwyn H. Gentry Forest Transect Dataset

"""

import os
import sys
import zipfile
import xlrd
from retriever.lib.templates import Script
from retriever.lib.models import Table
from retriever.lib.excel import Excel

VERSION = '1.0.1'

TAX_GROUPS = 9756 #9819

class main(Script):
    def __init__(self, **kwargs):
        Script.__init__(self, **kwargs)
        self.name = "Alwyn H. Gentry Forest Transect Dataset"
        self.shortname = "Gentry"
        self.urls = {"stems": "http://www.mobot.org/mobot/gentry/123/all_Excel.zip",
                     "sites": "http://www.ecologicaldata.org/sites/default/files/gentry_sites_data.txt",
                     "species": "",
                     "counts": ""}
        self.tags = ["Taxon > Plants", "Spatial Scale > Global",
                     "Data Type > Observational"]
        self.ref = "http://www.wlbcenter.org/gentry_data.htm"
        self.addendum = """Researchers who make use of the data in publications are requested to acknowledge Alwyn H. Gentry, the Missouri Botanical Garden, and collectors who assisted Gentry or contributed data for specific sites. It is also requested that a reprint of any publication making use of the Gentry Forest Transect Data be sent to:

Bruce E. Ponman
Missouri Botanical Garden
P.O. Box 299
St. Louis, MO 63166-0299
U.S.A. """

    def download(self, engine=None, debug=False):
        Script.download(self, engine, debug)

        self.engine.auto_create_table(Table("sites"), url=self.urls["sites"])
        self.engine.insert_data_from_url(self.urls["sites"])

        self.engine.download_file(self.urls["stems"], "all_Excel.zip")
        local_zip = zipfile.ZipFile(self.engine.format_filename("all_Excel.zip"))
        filelist = local_zip.namelist()
        local_zip.close()
        self.engine.download_files_from_archive(self.urls["stems"], filelist)

        filelist = [os.path.basename(filename) for filename in filelist]

        # Currently all_Excel.zip is missing CURUYUQU.xls
        # Download it separately and add it to the file list
        if not self.engine.find_file('CURUYUQU.xls'):
            self.engine.download_file("http://www.mobot.org/mobot/gentry/123/samerica/CURUYUQU.xls", "CURUYUQU.xls", clean_line_endings=False)
            filelist.append('CURUYUQU.xls')

        lines = []
        tax = []
        for filename in filelist:
            print "Extracting data from " + filename + "..."
            book = xlrd.open_workbook(self.engine.format_filename(filename))
            sh = book.sheet_by_index(0)
            rows = sh.nrows
            cn = {'stems': []}
            n = 0
            for colnum, c in enumerate(sh.row(0)):
                if not Excel.empty_cell(c):
                    cid = Excel.cell_value(c).lower()
                    # line number column is sometimes named differently
                    if cid in ["sub", "number"]:
                        cid = "line"
                    # the "number of individuals" column is named in various
                    # different ways; they always at least contain "nd"
                    if "nd" in cid:
                        cid = "count"
                    # in QUIAPACA.xls the "number of individuals" column is
                    # misnamed "STEMDBH" just like the stems columns, so weep
                    # for the state of scientific data and then fix manually
                    if filename == "QUIAPACA.xls" and colnum == 13:
                        cid = "count"

                    # if column is a stem, add it to the list of stems;
                    # otherwise, make note of the column name/number
                    if "stem" in cid or "dbh" in cid:
                        cn["stems"].append(n)
                    else:
                        cn[cid] = n
                n += 1
            # sometimes, a data file does not contain a liana or count column
            if not "liana" in cn.keys():
                cn["liana"] = -1
            if not "count" in cn.keys():
                cn["count"] = -1
            for i in range(1, rows):
                row = sh.row(i)
                cellcount = len(row)
                # make sure the row is real, not just empty cells
                if not all(Excel.empty_cell(cell) for cell in row):
                    try:
                        this_line = {}

                        def format_value(s):
                            s = Excel.cell_value(s)
                            return str(s).title().replace("\\", "/").replace('"', '')

                        # get the following information from the appropriate columns
                        for i in ["line", "family", "genus", "species",
                                  "liana", "count"]:
                            if cn[i] > -1:
                                this_line[i] = format_value(row[cn[i]])
                                if this_line[i] == '`':
                                    this_line[i] = 1

                        this_line["stems"] = [Excel.cell_value(row[c])
                                              for c in cn["stems"]
                                              if not Excel.empty_cell(row[c])]
                        this_line["site"] = filename[0:-4]

                        lines.append(this_line)

                        # Check how far the species is identified
                        full_id = 0
                        if len(this_line["species"]) < 3:
                            if len(this_line["genus"]) < 3:
                                id_level = "family"
                            else:
                                id_level = "genus"
                        else:
                            id_level = "species"
                            full_id = 1
                        tax.append((this_line["family"],
                                    this_line["genus"],
                                    this_line["species"].lower().replace('\\', '').replace('"', ''),
                                    id_level,
                                    str(full_id)))
                    except:
                        raise
                        pass

        tax = sorted(tax, key=lambda group: group[0] + " " + group[1] + " " + group[2])
        unique_tax = []
        tax_dict = dict()
        tax_count = 0

        # Get all unique families/genera/species
        for group in tax:
            if not (group in unique_tax):
                unique_tax.append(group)
                tax_count += 1
                tax_dict[group[0:3]] = tax_count
                if tax_count % 10 == 0:
                    msg = "Generating taxonomic groups: " + str(tax_count) + " / " + str(TAX_GROUPS)
                    sys.stdout.write(msg + "\b" * len(msg))
        print "Generating taxonomic groups: " + str(TAX_GROUPS) + " / " + str(TAX_GROUPS)

        # Create species table
        table = Table("species", delimiter=",")
        table.columns=[("species_id"            ,   ("pk-int",)    ),
                       ("family"                ,   ("char", )    ),
                       ("genus"                 ,   ("char", )    ),
                       ("species"               ,   ("char", )    ),
                       ("id_level"              ,   ("char", 10)    ),
                       ("full_id"               ,   ("bool",)       )]

        data = [','.join([str(tax_dict[group[:3]])] + ['"%s"' % g for g in group])
                for group in unique_tax]
        table.pk = 'species_id'
        table.contains_pk = True

        self.engine.table = table
        self.engine.create_table()
        self.engine.add_to_table(data)

        # Create stems table
        table = Table("stems", delimiter=",", contains_pk=False)
        table.columns=[("stem_id"               ,   ("pk-auto",)    ),
                       ("line"                  ,   ("int",)        ),
                       ("species_id"            ,   ("int",)        ),
                       ("site_code"             ,   ("char", 12)    ),
                       ("liana"                 ,   ("char", 10)    ),
                       ("stem"                  ,   ("double",)     )]
        stems = []
        counts = []
        for line in lines:
            try:
                liana = line["liana"]
            except KeyError:
                liana = ""
            species_info = [line["line"],
                            tax_dict[(line["family"],
                                      line["genus"],
                                      line["species"].lower())],
                            line["site"],
                            liana
                            ]
            try:
                counts.append([str(value) for value in species_info + [line["count"]]])
            except KeyError:
                pass

            for i in line["stems"]:
                stem = species_info + [i]
                stems.append([str(value) for value in stem])

        data = [','.join(stem) for stem in stems]
        self.engine.table = table
        self.engine.create_table()
        self.engine.add_to_table(data)

        # Create counts table
        table = Table("counts", delimiter=",", contains_pk=False)
        table.columns=[("count_id"              ,   ("pk-auto",)    ),
                       ("line"                  ,   ("int",)        ),
                       ("species_id"            ,   ("int",)        ),
                       ("site_code"             ,   ("char", 12)    ),
                       ("liana"                 ,   ("char", 10)    ),
                       ("count"                 ,   ("double",)     )]
        data = [','.join(count) for count in counts]
        self.engine.table = table
        self.engine.create_table()
        self.engine.add_to_table(data)

        return self.engine

SCRIPT = main()
ethanwhite commented 8 years ago

Thanks @ahhurlbert. Sorry for the delay. We're still working on this but running into some issues along the way. I can't reproduce this on Linux. I've been trying to reproduce it on Windows, but we've been running into issues with #32, which means we can't even get this far.

While we keep working on trying to reproduce can you give the current development version a try by running:

pip install git+https://github.com/weecology/retriever --upgrade
retriever reset all

Also, can you run which retriever from the command line and system('which retriever') from R and report what values they return?

ahhurlbert commented 8 years ago

Error on pip install

$ pip install git+https://github.com/weecology/retriever --upgrade
Collecting git+https://github.com/weecology/retriever
  Cloning https://github.com/weecology/retriever to c:\users\hurlbert\appdata\local\temp\pip-xsdwwn-build
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "c:\users\hurlbert\appdata\local\temp\pip-xsdwwn-build\setup.py", line 25, in <module>
        from __init__ import VERSION
      File "__init__.py", line 8, in <module>
        from builtins import str
    ImportError: No module named builtins

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in c:\users\hurlbert\appdata\local\temp\pip-xsdwwn-build\

so I did not run retriever reset all.

which retriever gives /c/Anaconda2/Scripts/retriever, but in R, system('which retriever') gives

Warning message:
running command 'which retriever' had status 127
ethanwhite commented 8 years ago

Agh, sorry about that. We recently broke our dependency free install apparently. We'll fix that today. In the mean time do this instead:

pip install future
pip install git+https://github.com/weecology/retriever --upgrade
retriever reset all

Also, can you report the output of these:

> system('retriever ls')
> system('ls')
> system('dir')

If you're available for our regularly meeting this afternoon we can also do more poking around then.

ethanwhite commented 8 years ago

For those following along at home, @ahhurlbert & I looked at this over a screen share & found that he was, at least, having issues with #32

OK, we know have #32 fixed... hopefully. Can you run

library(devtools)
install_github('ropensci/ecoretriever')
ecoretriever::datasets()

And see what happens?

ahhurlbert commented 8 years ago

I'm now back on my laptop on which I originally had this problem. I did this

pip install future
pip install git+https://github.com/weecology/retriever --upgrade
retriever reset all

and then in R get the same error as before:

>ecoretriever::datasets()
...
[78] "TAXONOMY"                                                                       
[79] "TIME SERIES"                                                                    
[80] "Failed to load script: gentry (C:/Users/hurlbert/Documents/.retriever/scripts/)"
ethanwhite commented 8 years ago

OK, can you report the output of the following two commands?

  1. From the command line run which retriever
  2. From inside R run shell('which retriever')
henrykironde commented 8 years ago

I am getting the results below after running these commands.

library(devtools)
install_github("ropensci/ecoretriever",force = TRUE)
library(ecoretriever)
ecoretriever::datasets()
session_info()

.all the datasets are shown upto 69 when they start to fail

.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
[69] "Failed to load script: bbs (C:/Users/Administrator/Documents/.retriever/scripts/)Failed to load script: bbs50stop (C:/Users/Administrator/Documents/.retriever/scripts/)Failed to load script: bioclim_2pt5 (C:/Users/Administrator/Documents/.retriever/scripts/)Failed to load script: fia (C:/Users/Administrator/Documents/.retriever/scripts/)Failed to load script: gentry (C:/Users/Administrator/Documents/.retriever/scripts/)Failed to load script: gwdd (C:/Users/Administrator/Documents/.retriever/scripts/)Failed to load script: leda (C:/Users/Administrator/Documents/.retriever/scripts/)Failed to load script: npn (C:/Users/Administrator/Documents/.retriever/scripts/)Failed to load script: prism_4km_monthly_modern (C:/Users/Administrator/Documents/.retriever/scripts/)Failed to load script: vertnet (C:/Users/Administrator/Documents/.retriever/scripts/)"
> session_info()
Session info ------------------------------------------------------------------------------------
 setting  value                       
 version  R version 3.2.0 (2015-04-16)
 system   x86_64, mingw32             
 ui       RStudio (0.99.902)          
 language (EN)                        
 collate  English_United States.1252  
 tz       America/New_York            
 date     2016-06-27                  

Packages ----------------------------------------------------------------------------------------
 package      * version date       source                                
 curl           0.9.7   2016-04-10 CRAN (R 3.2.5)                        
 devtools     * 1.12.0  2016-06-24 CRAN (R 3.2.5)                        
 digest         0.6.9   2016-01-08 CRAN (R 3.2.4)                        
 ecoretriever * 0.3.0   2016-06-28 Github (ropensci/ecoretriever@9025e1e)
 git2r          0.15.0  2016-05-11 CRAN (R 3.2.5)                        
 httr           1.2.0   2016-06-15 CRAN (R 3.2.0)                        
 knitr          1.13    2016-05-09 CRAN (R 3.2.0)                        
 memoise        1.0.0   2016-01-29 CRAN (R 3.2.5)                        
 R6             2.1.2   2016-01-26 CRAN (R 3.2.5)                        
 withr          1.0.1   2016-02-04 CRAN (R 3.2.5)                        
 yaml           2.1.13  2014-06-12 CRAN (R 3.2.5) 

shell('which retriever') /cygdrive/c/Python27/Scripts/retriever

ethanwhite commented 8 years ago

@henrykironde can you explain what you just posted a little bit? Is this the output and session info you have on your Windows machine?

henrykironde commented 8 years ago

Update: After retriever reset all the retriever on the same machine no scripts fail

henrykironde commented 8 years ago
library(ecoretriever)
ecoretriever::datasets()
 [1] "Available datasets : 47" ""                        "Adler2007"              
 [4] "AmnioteDB"               "Anderson2011"            "ArcticBreedBird"        
 [7] "AvianBodySize"           "BAAD"                    "BBS"                    
[10] "BBS50"                   "Bioclim"                 "CForBioData"            
[13] "Clark2006"               "DelMoral2010"            "eBirdOD"                
[16] "EltonTraits"             "FIA"                     "FishParasiteHosts"      
[19] "FrayJorge"               "Gentry"                  "GWDD"                   
[22] "HomeRanges"              "leaf_herbivory"          "leda"                   
[25] "MammalDIET"              "MammalLH"                "MammalMR2010"           
[28] "mammsupertree"           "MarineSize"              "Matter2014"             
[31] "MCDB"                    "McGlinn2010"             "MiscAbundanceDB"        
[34] "MoM2003"                 "NPN"                     "Palmer2007"             
[37] "Pantheria"               "Petraitis2008"           "PhytoplankonBiovolume"  
[40] "PlantTaxonomy"           "PortalMammals"           "PRISM"                  
[43] "Ramesh2010"              "SonoranPerennials"       "Steppe_plants_2013"     
[46] "TreeWesternGhats"        "vertnet"                 "Woods2009"              
[49] "Zachmann2010" 
ahhurlbert commented 8 years ago

Ok, I'm now back on my work machine and still doesn't work, but slightly different:

> ecoretriever::datasets()
 [1] "Available datasets : 45"                                                        
 [2] ""                                                                               
 [3] "Adler2007"                                                                      
 [4] "AmnioteDB"                                                                      
 [5] "Anderson2011"                                                                   
 [6] "ArcticBreedBird"                                                                
 [7] "AvianBodySize"                                                                  
 [8] "BAAD"                                                                           
 [9] "BBS"                                                                            
[10] "BBS50"                                                                          
[11] "Bioclim"                                                                        
[12] "CForBioData"                                                                    
[13] "Clark2006"                                                                      
[14] "DelMoral2010"                                                                   
[15] "eBirdOD"                                                                        
[16] "EltonTraits"                                                                    
[17] "FIA"                                                                            
[18] "FishParasiteHosts"                                                              
[19] "FrayJorge"                                                                      
[20] "GWDD"                                                                           
[21] "HomeRanges"                                                                     
[22] "leaf_herbivory"                                                                 
[23] "MammalDIET"                                                                     
[24] "MammalLH"                                                                       
[25] "MammalMR2010"                                                                   
[26] "mammsupertree"                                                                  
[27] "MarineSize"                                                                     
[28] "Matter2014"                                                                     
[29] "MCDB"                                                                           
[30] "McGlinn2010"                                                                    
[31] "MiscAbundanceDB"                                                                
[32] "MoM2003"                                                                        
[33] "NPN"                                                                            
[34] "Palmer2007"                                                                     
[35] "Pantheria"                                                                      
[36] "Petraitis2008"                                                                  
[37] "PhytoplankonBiovolume"                                                          
[38] "PlantTaxonomy"                                                                  
[39] "PortalMammals"                                                                  
[40] "PRISM"                                                                          
[41] "Ramesh2010"                                                                     
[42] "SonoranPerennials"                                                              
[43] "Steppe_plants_2013"                                                             
[44] "TreeWesternGhats"                                                               
[45] "vertnet"                                                                        
[46] "Woods2009"                                                                      
[47] "Zachmann2010"                                                                   
[48] "Failed to load script: gentry (C:/Users/hurlbert/Documents/.retriever/scripts/)"
[49] "Exception: cannot import name ScriptTest "  

In both cases when I type retriever reset all and it asks if I want to proceed I type 'y' and nothing happens. I seem to recall in the past that I would see progress of redownloading scripts, or is that a faulty memory.

BTW,

$ which retriever
/c/Anaconda2/Scripts/retriever

and

> shell("which retriever")
'which' is not recognized as an internal or external command,
operable program or batch file.
Warning messages:
1: running command 'C:\Windows\system32\cmd.exe /c which retriever' had status 1 
2: In shell("which retriever") :
  'which retriever' execution failed with error code 1
> system("which retriever")
Warning message:
running command 'which retriever' had status 127 

We can troubleshoot further over our 2pm chat...

henrykironde commented 8 years ago

you can do a reset manually by deleting the entire C:/Users/hurlbert/Documents/.retriever

ahhurlbert commented 8 years ago

Had to manually delete the Ecoretriever folder in Program Files as well as the .retriever folder mentioned above. Now, all is well!

ethanwhite commented 8 years ago

Adding a little more information in case anyone else encounters this. There were two installations of the retriever, one using the Windows installer and one using pip. The command line was finding the pip install, but the R package was finding the older Windows installer version. Once we removed the older version (having already fixed #32) everything works as expected.

dmcglinn commented 8 years ago

Big thanks to @ethanwhite for following this up and also to @ahhurlbert for reporting! I'll try to whip up #69 before CRAN submission stay tuned for future PR

ahhurlbert commented 8 years ago

I'm having a similar problem to before with reading datasets using ecoretriever in R. As suggested above I have

1) deleted all existing versions of EcoDataRetriever that I know of including folders in Program Files, Program Files (x86), /c/users/hurlbert/.retriever, /c/users/hurlbert/appdata/local/continuum/anaconda/scripts/retriever, and /c/users/hurlbert/documents/R/win-library/ecoretriever 2) reinstalled ecoretriever from scratch

pip install future
pip install git+https://github.com/weecology/retriever --upgrade

after which I get

$ pip install git+https://github.com/weecology/retriever --upgrade
Downloading/unpacking git+https://github.com/weecology/retriever
  Cloning https://github.com/weecology/retriever to c:\users\hurlbert\appdata\lo
cal\temp\pip-zjnivb-build
  Running setup.py (path:c:\users\hurlbert\appdata\local\temp\pip-zjnivb-build\s
etup.py) egg_info for package from git+https://github.com/weecology/retriever
    C:\Users\hurlbert\AppData\Local\Continuum\Anaconda\lib\distutils\dist.py:267
: UserWarning: Unknown distribution option: 'console'
      warnings.warn(msg)
    C:\Users\hurlbert\AppData\Local\Continuum\Anaconda\lib\distutils\dist.py:267
: UserWarning: Unknown distribution option: 'app'
      warnings.warn(msg)
    C:\Users\hurlbert\AppData\Local\Continuum\Anaconda\lib\distutils\dist.py:267
: UserWarning: Unknown distribution option: 'zipfile'
      warnings.warn(msg)

    Compiling retriever scripts...
    Adler2007 is supported by Retriever version 2.0.0-dev
    Current version is v2.0.0-dev

...<all the scripts load>...

    Zachmann2010 is supported by Retriever version 2.0.0-dev
    Current version is v2.0.0-dev
    done.
Requirement already up-to-date: xlrd in c:\users\hurlbert\appdata\local\continuu
m\anaconda\lib\site-packages (from retriever==2.0.0-de)
Requirement already up-to-date: future in c:\users\hurlbert\appdata\local\contin
uum\anaconda\lib\site-packages (from retriever==2.0.0-de)
Installing collected packages: retriever
  Found existing installation: retriever 2.0.0-de
    Uninstalling retriever:
      Successfully uninstalled retriever
  Running setup.py install for retriever
    C:\Users\hurlbert\AppData\Local\Continuum\Anaconda\lib\distutils\dist.py:267
: UserWarning: Unknown distribution option: 'console'
      warnings.warn(msg)
    C:\Users\hurlbert\AppData\Local\Continuum\Anaconda\lib\distutils\dist.py:267
: UserWarning: Unknown distribution option: 'app'
      warnings.warn(msg)
    C:\Users\hurlbert\AppData\Local\Continuum\Anaconda\lib\distutils\dist.py:267
: UserWarning: Unknown distribution option: 'zipfile'
      warnings.warn(msg)

    Installing retriever-script.py script to C:\Users\hurlbert\AppData\Local\Con
tinuum\Anaconda\Scripts
    Installing retriever.exe script to C:\Users\hurlbert\AppData\Local\Continuum
\Anaconda\Scripts
    Compiling retriever scripts...
    Adler2007 is supported by Retriever version 2.0.0-dev

...<all the scripts load AGAIN>...

    Zachmann2010 is supported by Retriever version 2.0.0-dev
    Current version is v2.0.0-dev
    done.
Successfully installed retriever
Cleaning up...

So it looks like it's not a clean install, or it installs, uninstalls, and then reinstalls.

In any case, which and version are:

$ which retriever
/c/Users/hurlbert/AppData/Local/Continuum/Anaconda/Scripts/retriever

$ retriever --version
v2.0.0-dev

I ran retriever reset all and said yes, with no complaints.

Finally, in RStudio (running R 3.3.1) I get:

> library(devtools)

Attaching package: ‘devtools’

The following object is masked from ‘package:ecoretriever’:

    install

> install_github("ropensci/ecoretriever",force = TRUE)
Downloading GitHub repo ropensci/ecoretriever@master
from URL https://api.github.com/repos/ropensci/ecoretriever/zipball/master
Installing ecoretriever
"C:/PROGRA~1/R/R-33~1.1/bin/x64/R" --no-site-file --no-environ --no-save --no-restore --quiet  \
  CMD INSTALL  \
  "C:/Users/hurlbert/AppData/Local/Temp/RtmpQnXoNK/devtools111441a842c5/ropensci-ecoretriever-9025e1e"  \
  --library="C:/Users/hurlbert/Documents/R/win-library/3.3" --install-tests 

* installing *source* package 'ecoretriever' ...
** R
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
*** arch - i386
*** arch - x64
* DONE (ecoretriever)
Reloading installed ecoretriever

  Use get_updates() to download the most recent release of download scripts

    New to ecoretriever? Examples at
      https://github.com/ropensci/ecoretriever/
      Use citation(package='ecoretriever') for the package citation

Use suppressPackageStartupMessages() to suppress these messages in the future

Attaching package: ‘ecoretriever’

The following object is masked from ‘package:devtools’:

    install

> library(ecoretriever)
> ecoretriever::datasets()
 [1] "No scripts are currently available. Updating scripts now..."                                                                                               
 [2] "Traceback (most recent call last):"                                                                                                                        
 [3] "  File \"C:\\Users\\hurlbert\\AppData\\Local\\Continuum\\Anaconda\\Scripts\\retriever-script.py\", line 9, in <module>"                                    
 [4] "    load_entry_point('retriever==2.0.0-de', 'console_scripts', 'retriever')()"                                                                             
 [5] "  File \"C:\\Users\\hurlbert\\AppData\\Local\\Continuum\\Anaconda\\lib\\site-packages\\retriever\\__main__.py\", line 122, in main"                        
 [6] "    check_for_updates()"                                                                                                                                   
 [7] "  File \"C:\\Users\\hurlbert\\AppData\\Local\\Continuum\\Anaconda\\lib\\site-packages\\retriever\\lib\\repository.py\", line 51, in check_for_updates"     
 [8] "    init.run()"                                                                                                                                            
 [9] "  File \"C:\\Users\\hurlbert\\AppData\\Local\\Continuum\\Anaconda\\lib\\site-packages\\retriever\\lib\\repository.py\", line 96, in run"                   
[10] "    version_file = urllib.request.urlopen(REPOSITORY + \"version.txt\")"                                                                                   
[11] "  File \"C:\\Users\\hurlbert\\AppData\\Local\\Continuum\\Anaconda\\lib\\site-packages\\future\\backports\\urllib\\request.py\", line 171, in urlopen"      
[12] "    return opener.open(url, data, timeout)"                                                                                                                
[13] "  File \"C:\\Users\\hurlbert\\AppData\\Local\\Continuum\\Anaconda\\lib\\site-packages\\future\\backports\\urllib\\request.py\", line 494, in open"         
[14] "    response = self._open(req, data)"                                                                                                                      
[15] "  File \"C:\\Users\\hurlbert\\AppData\\Local\\Continuum\\Anaconda\\lib\\site-packages\\future\\backports\\urllib\\request.py\", line 517, in _open"        
[16] "    'unknown_open', req)"                                                                                                                                  
[17] "  File \"C:\\Users\\hurlbert\\AppData\\Local\\Continuum\\Anaconda\\lib\\site-packages\\future\\backports\\urllib\\request.py\", line 466, in _call_chain"  
[18] "    result = func(*args)"                                                                                                                                  
[19] "  File \"C:\\Users\\hurlbert\\AppData\\Local\\Continuum\\Anaconda\\lib\\site-packages\\future\\backports\\urllib\\request.py\", line 1353, in unknown_open"
[20] "    raise URLError('unknown url type: %s' % type)"                                                                                                         
[21] "future.backports.urllib.error.URLError: <urlopen error unknown url type: https>"                                                                           
attr(,"status")
[1] 1
Warning message:
running command 'C:\Windows\system32\cmd.exe /c retriever ls' had status 1 

Do I still have ecoretriever files from a previous install hiding out somewhere on my machine (Windows 8) that are messing things up?

ethanwhite commented 8 years ago

This is a separate issue from the previous one. Let's do the following:

  1. Try running retriever update from the command line. If that errors then this is an issue with the retriever itself. Open a new issue in the retriever repository.
  2. If it works from the command line trying running it from R (not RStudio) and then open a new issue here and add information on whether it fails in both R and RStudio or just in RStudio. Also run > system("retriever --version") in both R and RStudio and report what it returns.

From the looks of things I suspect it's going to be a core retriever issue.

ethanwhite commented 8 years ago

Closing this issue again since this is a new problem.

ahhurlbert commented 8 years ago

Unfortunately, the problem is on my laptop at home, so I won't be able to test until tonight. However, I did run retriever update when messing around before and definitely got some kind of unhappy error message.

Will follow up and post a separate issue tonight.

ethanwhite commented 8 years ago

Thanks Allen.