search

Rblp / Rblpapi

R package interfacing the Bloomberg API from https://www.bloomberglabs.com/api/
Other
166 stars 75 forks source link

Issues with Latest Tarball #294

Closed samvaltenbergs closed 5 years ago

samvaltenbergs commented 5 years ago

I doubt this is an issues with Rblpapi itself, but I'm not having luck on the forums/web finding a solution to this so hopefully someone can help. I am trying to download the latest Rblpapi from CRAN but when my RStudio session tries to install it from the tarball it gives me the following error. The issue seems to be with extracting the tarball. 

* installing *source* package 'Rblpapi' ...
** package 'Rblpapi' successfully unpacked and MD5 sums checked

   **********************************************
   WARNING: this package has a configure script
         It probably needs manual configuration
   **********************************************

** libs

*** arch - i386

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Exiting with failure status due to previous errors
make: *** [blpLibrary] Error 2
Warning: running command 'make -f "Makevars.win" -f "C:/PROGRA~1/R/R-33~1.3/etc/i386/Makeconf" -f "C:/PROGRA~1/R/R-33~1.3/share/make/winshlib.mk" CXX='$(CXX1X) $(CXX1XSTD)' CXXFLAGS='$(CXX1XFLAGS)' CXXPICFLAGS='$(CXX1XPICFLAGS)' SHLIB_LDFLAGS='$(SHLIB_CXX1XLDFLAGS)' SHLIB_LD='$(SHLIB_CXX1XLD)' SHLIB="Rblpapi.dll" OBJECTS="RcppExports.o authenticate.o bdh.o bdp.o bds.o beqs.o blpConnect.o blpVersion.o blpapi_utils.o bsrch.o fieldsearch.o getBars.o getFieldInfo.o getTicks.o lookup.o subscribe.o"' had status 2
ERROR: compilation failed for package 'Rblpapi'
* removing 'C:/Program Files/R/R-3.3.3/library/Rblpapi'
Warning in install.packages :
  running command '"C:/PROGRA~1/R/R-33~1.3/bin/x64/R" CMD INSTALL -l "C:\Program Files\R\R-3.3.3\library" "c:/temp/Rblpapi_0.3.10.tar.gz"' had status 1
Warning in install.packages :
  installation of package ‘c:/temp/Rblpapi_0.3.10.tar.gz’ had non-zero exit status

My RStudio details:

platform       x86_64-w64-mingw32          
arch           x86_64                      
os             mingw32                     
system         x86_64, mingw32             
status                                     
major          3                           
minor          3.3                         
year           2017                        
month          03                          
day            06                          
svn rev        72310                       
language       R                           
version.string R version 3.3.3 (2017-03-06)
nickname       Another Canoe

Any help would be appreciated.

eddelbuettel commented 5 years ago

The line

gzip: stdin: not in gzip format

suggests that not even the download worked. Maybe you have proxy issues. We can't tell. Try to decompose everything to the individual steps and try to drill down what works or fails.

samvaltenbergs commented 5 years ago

THanks, will do. Proxy issues wouldn’t surprise me (sadly…).

On Jul 19, 2019, at 1:16 PM, Dirk Eddelbuettel notifications@github.com wrote:

The line

gzip: stdin: not in gzip format suggests that not even the download worked. Maybe you have proxy issues. We can't tell. Try to decompose everything to the individual steps and try to drill down what works or fails.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Rblp/Rblpapi/issues/294?email_source=notifications&email_token=AKN3PO4KYQBHF2FLEY5DFL3QAIOLPA5CNFSM4IFKX7F2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2MU5HA#issuecomment-513363612, or mute the thread https://github.com/notifications/unsubscribe-auth/AKN3PO6MRTS5HPKOIE6RJU3QAIOLPANCNFSM4IFKX7FQ.

eddelbuettel commented 5 years ago

@samvaltenbergs Thanks for coming back to this for closure. And yes, suspected "bugs" are sometimes merely local setup issues. It happens.

samvaltenbergs commented 5 years ago

Sorry guys... I did some more experimenting. At the end of the install.packages() call upon failure it says:

The downloaded source packages are in
    ‘C:\Users\samv\AppData\Local\Temp\RtmpSclx6J\downloaded_packages’

I can see the tarball in this directory. I can also untar() it successfully and see that the untar() call unpacks the directory contents as it should.

Any other ideas why install.packages() would complain about the tarball even though I can see that install.packages() downloads the tarball correctly and untar() can correctly unpack it?

eddelbuettel commented 5 years ago

We do not, and we are not on your box. I invite you to debug this locally -- only you can.

The key still is that

i) We get a source package from CRAN ii) The source package gets unpackage iii) As first step of the installation process the configure script runs (if you're on Unix, not in your case) iv) Next the src/Makevars{,.win} script run, for you on Windoze it is Makevars.win which runs a few steps

Look at those and see if/why/how they fail.

samvaltenbergs commented 5 years ago

For step "ii)" above, how can one determine the function that R is trying to use to unpackage the tarball? I can untar() it from the command line in RStudio, and can also unpack it from the Windows command line without any issues using tar xvf Rblpapi_0.3.10.tar.gz. However, RStudio complains about gzip: stdin: not in gzip format when unpacking as part of install.packages().

eddelbuettel commented 5 years ago

R CMD INSTALL .... does it. Invariable, it will call some code in (base R) packages tools or utils.

If it fails in RStudio, maybe we try without RStudio?

samvaltenbergs commented 5 years ago

Tried from the standard R console (not RStudio) and the behavior is exactly the same.

I currently have Rblpapi 0.3.8 but I need 0.3.10 for B-Pipe authentication. 0.3.8 was installed from binary, so I never encountered this issue before. But if I try to install 0.3.8 from source (type = "source") I receive the same issue: gzip: stdin: not in gzip format. Same for 0.3.9... So type = "source" gives me the same gzip error for 0.3.8, 0.3.9 and 0.3.10.

As a check, I also tried to install a handful of other randomly selected packages with type = "source" to double check I can actually install packages from source. I didn't receive any errors installing ggrepel, ggplot2, tidyr and knitr from source, I only seem to have this issue with the Rblpapi tarball.

Very confused...

eddelbuettel commented 5 years ago

No idea. Looks fine from here:

edd@rob:/tmp$ wget https://cloud.r-project.org/src/contrib/Rblpapi_0.3.10.tar.gz
--2019-07-23 12:14:55--  https://cloud.r-project.org/src/contrib/Rblpapi_0.3.10.tar.gz
Resolving cloud.r-project.org (cloud.r-project.org)... 13.226.15.93, 13.226.15.114, 13.226.15.118, ...
Connecting to cloud.r-project.org (cloud.r-project.org)|13.226.15.93|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 171617 (168K) [application/x-gzip]
Saving to: ‘Rblpapi_0.3.10.tar.gz’

Rblpapi_0.3.10.tar.gz                               100%[===================================================================================================================>] 167.59K  --.-KB/s    in 0.04s

2019-07-23 12:14:56 (4.47 MB/s) - ‘Rblpapi_0.3.10.tar.gz’ saved [171617/171617]

edd@rob:/tmp$ tar tvzf Rblpapi_0.3.10.tar.gz | head
drwxr-xr-x ligges/users      0 2019-04-07 05:32 Rblpapi/
drwxr-xr-x ligges/users      0 2019-04-02 06:56 Rblpapi/inst/
-rw-r--r-- ligges/users   1047 2014-11-26 15:35 Rblpapi/inst/License.txt
-rw-r--r-- ligges/users   8256 2019-04-02 06:56 Rblpapi/inst/NEWS.Rd
drwxr-xr-x ligges/users      0 2019-04-02 06:56 Rblpapi/inst/auto/
-rw-r--r-- ligges/users    107 2019-04-02 06:56 Rblpapi/inst/auto/NEWS.el
drwxr-xr-x ligges/users      0 2019-04-02 06:56 Rblpapi/inst/doc/
-rw-r--r-- ligges/users   1056 2019-04-02 06:56 Rblpapi/inst/doc/rblpapi-intro.R
-rw-r--r-- ligges/users   5500 2019-02-21 07:38 Rblpapi/inst/doc/rblpapi-intro.Rmd
-rw-r--r-- ligges/users  16630 2019-04-02 06:56 Rblpapi/inst/doc/rblpapi-intro.html
edd@rob:/tmp$ md5sum Rblpapi_0.3.10.tar.gz
57cf8002f281b4782872188d7efd7f04  Rblpapi_0.3.10.tar.gz
edd@rob:/tmp$
samvaltenbergs commented 5 years ago

I tried unpacking the tarball myself and installing using devtools:

install_local("C:/Users/samv/AppData/Local/Temp/Rtmpa29IfN/downloaded_packages/Rblpapi")

Same issue... Turns out the issue isn't with the main package tarball, but the tarballs being pulled down as part of the Makevars.win file in the /src directory. I downloaded blpLibrary.tar.gz and blpHeaders.tar.gz files manually and tried to unpack them one at a time.

From the RStudio console:

untar("c:/users/samv/downloads/blpLibrary.tar.gz")
    /usr/bin/tar: This does not look like a tar archive
    /usr/bin/tar: Skipping to next header
    /usr/bin/tar: Exiting with failure status due to previous errors
    Warning messages:
    1: running command 'tar.exe -zxf "c:/users/samv/downloads/blpLibrary.tar.gz"' had status 2 
    2: In untar("c:/users/samv/downloads/blpLibrary.tar.gz") :
      ‘tar.exe -zxf "c:/users/samv/downloads/blpLibrary.tar.gz"’ returned error code 2

untar("c:/users/samv/downloads/blpHeaders.tar.gz")
    /usr/bin/tar: This does not look like a tar archive
    /usr/bin/tar: Skipping to next header
    /usr/bin/tar: Exiting with failure status due to previous errors
    Warning messages:
    1: running command 'tar.exe -zxf "c:/users/samv/downloads/blpHeaders.tar.gz"' had status 2 
    2: In untar("c:/users/samv/downloads/blpHeaders.tar.gz") :
      ‘tar.exe -zxf "c:/users/samv/downloads/blpHeaders.tar.gz"’ returned error code 2

And also tried from the Windows console:

c:\Users\samv\Downloads>tar xvf blpHeaders.tar.gz
    tar: This does not look like a tar archive
    tar: Skipping to next header
    tar: Exiting with failure status due to previous errors

c:\Users\samv\Downloads>tar xvf blpLibrary.tar.gz
    tar: This does not look like a tar archive
    tar: Skipping to next header
    tar: Exiting with failure status due to previous errors

So it seems the issue is with these two compressed files... I also tried with 7-Zip (which is installed on my workstation) and it can't unpack the tar file inside the tar.gz file, errors out.

Ideas?

eddelbuettel commented 5 years ago

I am only going to say this one more time: None of what we have written suggests or recommends additional obfuscation layers. If you hit a wall using RStudio, devtools or ... then you need to work out why.

The package is at CRAN. And builds and checks fine. That is the quality standard we measure against.

We try to help you but we're at the same stage we've been at for a while: your machine seems to download differently from others. Maybe it is between you and GitHub (that is where the other tarballs comes from). We cannot tell.

Good luck.

randomee commented 5 years ago

It does seem like you have a download issue. Can you check the md5sum or sha256sum?

$ curl -s -k -L -O https://github.com/Rblp/blp/raw/master/win64/blpLibrary.tar.gz
$ curl -s -k -L -O https://github.com/Rblp/blp/raw/master/headers/windows/blpHeaders.tar.gz
$ md5sum blp*.gz
89cff770c4d66533c0ecd89bcee1d441  blpHeaders.tar.gz
51dd870e05d9a395850fc10e8fabdff8  blpLibrary.tar.gz
$ sha256sum blp*.gz
bb3d90753be47467f977215327da9b038f33fb24312ae8bf00893d98f6b5b6b5  blpHeaders.tar.gz
472dfa7b2da6b43cecc753b5a487051b55562ab5acf4763d1490b8b0498f6194  blpLibrary.tar.gz

If it's not downloading properly via Makevars.win/curl, then it's not hard to manually download/place them.

samvaltenbergs commented 5 years ago

Yes, the downloading of the blpHeaders.tar.gz and blpLibrary.tar.gz files was indeed the issue. I sit behind a pretty restrictive corporate proxy and firewall, so having my Windows workstation use curl to download the two tarballs adds a layer of complexity to installing 0.3.10 from source. Some back-and-forth with the network infrastructure team at my employer finally got things fixes...

I'm no expert, but I wonder if it would be cleaner/less prone to proxy/firewall issues if the Bloomberg headers and library files were packaged with the main Rblpapi tarball instead of being stored separately on GitHub?

Either way, thank you all for working through this with me, appreciate the time and effort and sorry for the runaround!

eddelbuettel commented 5 years ago

No, it is not possible to include them (as CRAN cannot hold cloded-source code like this written by Bloomberg and released sans source) which is why they are where your found them. We didn't set this up just to torture you. We this up to automate installation as much as possible and while I as a co-author of that code may be a little partial it was by and large a roaring success. It allowed the package to be on CRAN -- which got it into the hands of more users than befire.

And those of us who had their heads bent from looking at compiler error messages and alike told you weeks ago that this looked like a network / proxy issue at your end. Anyway -- water under the bridge now. Glad you have it working.