search

eguidotti / RblDataLicense

R Interface to Bloomberg Data License
http://rbldatalicense.eguidotti.com
GNU General Public License v3.0
14 stars 3 forks source link

Unable to download Bloomberg file #3

Closed Vannies closed 4 years ago

Vannies commented 4 years ago

Trying the All in one example. I can see the req file uploaded, Checking if file Rbl_xxxxx.out.gz is availale and then the message - file not yet available which run till timeout.

Using RblFiles() the file is listed, trying RblDownload() to extract the file I get the same message - file not yet available which run till timeout.

Tried several different days, diff securities still not able to download the file from Bloomberg.

eguidotti commented 4 years ago

Hello @Vannies, you may try the following code to help debugging:

# replace with your filename
file <- "Rbl_xxxxx.out.gz"

# download
url  <- RblUrl()
tmp  <- tempfile()
utils::download.file(paste(url, file, sep='/'), destfile = tmp)

This is basically the code used by the RblDownload() function. It's likely you will get some error messages, hopefully useful to identify the problem. The issue might be due to some specific configuration of your license, as the package seems to work fine for most users. If this is not the case, please share your findings and I'll be happy to improve the package. Thanks

Vannies commented 4 years ago

Good day Emanuele

Thank you for your reply. I did try your suggestion and received the following error message:

Error in utils::download.file(paste(url, file, sep = "/"), destfile = tmp) : scheme not supported in URL 'sftp://dlxxxxxx:zzzzzzzzzzzzzzzz@dlsftp.bloomberg.com:30206/Rbl_0624072213.out'

Regards

From: Emanuele Guidotti [mailto:notifications@github.com] Sent: 20 June 2020 07:27 PM To: emanuele-guidotti/RblDataLicense RblDataLicense@noreply.github.com Cc: Johann VanderMerwe johann.vandermerwe@resbank.co.za; Mention mention@noreply.github.com Subject: Re: [emanuele-guidotti/RblDataLicense] Unable to download Bloomberg file (#3)

Hello @Vannieshttps://github.com/Vannies, you may try the following code to help debugging:

replace with your filename

file <- "Rbl_xxxxx.out.gz"

download

url <- RblUrl()

tmp <- tempfile()

utils::download.file(paste(url, file, sep='/'), destfile = tmp)

This is basically the code used by the RblDownload() function. It's likely you will get some error messages, hopefully useful to identify the problem. The issue might be due to some specific configuration of your license, as the package seems to work fine for most users. If this is not the case, please share your findings and I'll be happy to improve the package. Thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/emanuele-guidotti/RblDataLicense/issues/3#issuecomment-647023990, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQAHEPVFQ6MGNZ6HCZAJT53RXTWNLANCNFSM4OCTMY4A.

--------------------------------------------------------- Disclaimer --------------------------------------------------------- Important Notice: This e-mail is subject to the e-mail disclaimer of the South African Reserve Bank, which can be viewed at: http://www.resbank.co.za/Disclaimer/Pages/SARB-Disclaimer.aspx Should you be unable to access the link provided, kindly send an email to BSTD-ICT-ServiceDesk@resbank.co.za --------------------------------------------------------- Disclaimer ---------------------------------------------------------

eguidotti commented 4 years ago

Hello @Vannies, it's a bit weird. May you try to run sessionInfo() and send the (formatted) output here? So that I can check your R version and OS. Please make sure you have libcurl installed, as requested by the RCurl package used by RblDataLicense. You may also want to try downloading the file with the following code and let me know what happens:

# replace with your filename
file <- "Rbl_xxxxx.out.gz"

# download
url  <- RblUrl()
x    <- getURL(paste(url, file, sep='/'))

Please feel free to try downloading that file with other commands/code and let me know what works best for you. Thanks

Vannies commented 4 years ago

Good day Emanuele

Thank you for the response.

I will run your request as soon as my network guys respond to a query I made related to the url in question.

Regards Johann van der Merwe

From: Emanuele Guidotti [mailto:notifications@github.com] Sent: 24 June 2020 12:35 PM To: emanuele-guidotti/RblDataLicense RblDataLicense@noreply.github.com Cc: Johann VanderMerwe johann.vandermerwe@resbank.co.za; Mention mention@noreply.github.com Subject: Re: [emanuele-guidotti/RblDataLicense] Unable to download Bloomberg file (#3)

Hello @Vannieshttps://github.com/Vannies, it's a bit weird. May you try to run sessionInfo() and send the (formatted) output here? So that I can check your R version and OS. Please make sure you have libcurl installed, as requested by the RCurl packagehttps://cran.r-project.org/package=RCurl used by RblDataLicense. You may also want to try downloading the file with the following code and let me know what happens:

replace with your filename

file <- "Rbl_xxxxx.out.gz"

download

url <- RblUrl()

x <- getURL(paste(url, file, sep='/'))

Please feel free to try downloading that file with other commands/code and let me know what works best for you. Thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/emanuele-guidotti/RblDataLicense/issues/3#issuecomment-648740472, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQAHEPUQG2D2NQW4W2P7OO3RYHJGBANCNFSM4OCTMY4A.

--------------------------------------------------------- Disclaimer --------------------------------------------------------- Important Notice: This e-mail is subject to the e-mail disclaimer of the South African Reserve Bank, which can be viewed at: http://www.resbank.co.za/Disclaimer/Pages/SARB-Disclaimer.aspx Should you be unable to access the link provided, kindly send an email to BSTD-ICT-ServiceDesk@resbank.co.za --------------------------------------------------------- Disclaimer ---------------------------------------------------------

Vannies commented 4 years ago

Good day Emanuele

Apologies for my late response, I’ve been booked of sick twice since your last e-mail. Kindly see the attached document with the results of your requests in the below e-mail. I did the checks on two platforms, my laptop and a server I have access to.

On a separate occasion I was able to access the Bloomberg sftp server manually and copy a RBL_...... file.

Regards Johann van der Merwe

From: Emanuele Guidotti [mailto:notifications@github.com] Sent: 24 June 2020 12:35 PM To: emanuele-guidotti/RblDataLicense RblDataLicense@noreply.github.com Cc: Johann VanderMerwe johann.vandermerwe@resbank.co.za; Mention mention@noreply.github.com Subject: Re: [emanuele-guidotti/RblDataLicense] Unable to download Bloomberg file (#3)

Hello @Vannieshttps://github.com/Vannies, it's a bit weird. May you try to run sessionInfo() and send the (formatted) output here? So that I can check your R version and OS. Please make sure you have libcurl installed, as requested by the RCurl packagehttps://cran.r-project.org/package=RCurl used by RblDataLicense. You may also want to try downloading the file with the following code and let me know what happens:

replace with your filename

file <- "Rbl_xxxxx.out.gz"

download

url <- RblUrl()

x <- getURL(paste(url, file, sep='/'))

Please feel free to try downloading that file with other commands/code and let me know what works best for you. Thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/emanuele-guidotti/RblDataLicense/issues/3#issuecomment-648740472, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQAHEPUQG2D2NQW4W2P7OO3RYHJGBANCNFSM4OCTMY4A.

--------------------------------------------------------- Disclaimer --------------------------------------------------------- Important Notice: This e-mail is subject to the e-mail disclaimer of the South African Reserve Bank, which can be viewed at: http://www.resbank.co.za/Disclaimer/Pages/SARB-Disclaimer.aspx Should you be unable to access the link provided, kindly send an email to BSTD-ICT-ServiceDesk@resbank.co.za --------------------------------------------------------- Disclaimer ---------------------------------------------------------

Vannies commented 4 years ago

Good day Emanuele

As per your request:

A. Results of sessionInfo(): R version 3.6.3 (2020-02-29) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows Server x64 (build 17763)

B. Results of

b <- libcurlVersion() "7.59.0"

C. Results of

download

url <- RblUrl() x <- getURL(paste(url, file, sep='/'))

"begin 600 Rbl_0720100239_1.out.gz\nMA<@6[C4K9CA,6.YM#>D+YI#)54R\"TXFTYX<H0ROU=!IS B[_P5<)^175+O\nM-C\\.^DH6SD[7L6R,X81K6CG'6+7)$'UV37^ ?&DS<;,<LUE=OS(SFZL@? L\nM!UPA7C24 30DH:FK!$5P'>ML&+R,$74PXK>I9ZCXA& B8XG$NO'\ 8&WQ%\nM,6X2P)SA7?\"F]UZ2W#VX/M(>^:V5#\"RS\"%N<:D\"-#)9CU-(#C CCA.YG_9\nMCV!7/UP@0,V0T(&Z^]Z3:\"IDF.\$6(EH]V0)54XYVVF40\B^0+]T4K;5A\4\nMP7&TX7 KF=O2UD;4^G7/G8(+5->2;E#>;\"-LR60F4Y[@IF#PWM+)5\"J\" IE\n:KW#.A\"CJA @M0#2 !(\"0[E-X][W;\"=FN=, \n \nend\n"

Regards Johann van der Merwe

From: Emanuele Guidotti [mailto:notifications@github.com] Sent: 24 June 2020 12:35 PM To: emanuele-guidotti/RblDataLicense RblDataLicense@noreply.github.com<mailto:RblDataLicense@noreply.github.com> Cc: Johann VanderMerwe johann.vandermerwe@resbank.co.za<mailto:johann.vandermerwe@resbank.co.za>; Mention mention@noreply.github.com<mailto:mention@noreply.github.com> Subject: Re: [emanuele-guidotti/RblDataLicense] Unable to download Bloomberg file (#3)

Hello @Vannieshttps://github.com/Vannies, it's a bit weird. May you try to run sessionInfo() and send the (formatted) output here? So that I can check your R version and OS. Please make sure you have libcurl installed, as requested by the RCurl packagehttps://cran.r-project.org/package=RCurl used by RblDataLicense. You may also want to try downloading the file with the following code and let me know what happens:

replace with your filename

file <- "Rbl_xxxxx.out.gz"

download

url <- RblUrl()

x <- getURL(paste(url, file, sep='/'))

Please feel free to try downloading that file with other commands/code and let me know what works best for you. Thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/emanuele-guidotti/RblDataLicense/issues/3#issuecomment-648740472, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQAHEPUQG2D2NQW4W2P7OO3RYHJGBANCNFSM4OCTMY4A.

--------------------------------------------------------- Disclaimer --------------------------------------------------------- Important Notice: This e-mail is subject to the e-mail disclaimer of the South African Reserve Bank, which can be viewed at: http://www.resbank.co.za/Disclaimer/Pages/SARB-Disclaimer.aspx Should you be unable to access the link provided, kindly send an email to BSTD-ICT-ServiceDesk@resbank.co.za --------------------------------------------------------- Disclaimer ---------------------------------------------------------

eguidotti commented 4 years ago

Hello @Vannies, thank you for the update. It looks like you are using an old data license still using encrypted files. The issue seems similar to #2 Would you mind having a look there?

I think it should be possible to remove the encryption by contacting Bloomberg support. But please note that I'm not related to Bloomberg in any way, so I might be wrong.

If this option doesn't work for you, I may consider to have the package supporting encrypted files. But I'm afraid I need help from someone at bloomberg for that. So it would be rather complicated I guess

Vannies commented 4 years ago

Hallo Emanuele

Thank you for the reply.

I’m in conversation with Bloomberg regarding the encryption and will advise as soon as I have an answer.

For interest sake. Will the encryption stop RblDownload to “see” a file and even download the file? Why I ask is that RblFiles is able to see what files are ready to be downloaded and with RblDownload the message is the file is not yet available.

Regards Johann van der Merwe

From: Emanuele Guidotti [mailto:notifications@github.com] Sent: 27 July 2020 08:16 PM To: eguidotti/RblDataLicense RblDataLicense@noreply.github.com Cc: Johann VanderMerwe johann.vandermerwe@resbank.co.za; Mention mention@noreply.github.com Subject: Re: [eguidotti/RblDataLicense] Unable to download Bloomberg file (#3)

Hello @Vannieshttps://github.com/Vannies, thank you for the update. It looks like you are using an old data license still using encrypted files. The issue seems similar to #2https://github.com/eguidotti/RblDataLicense/issues/2 Would you mind having a look there?

I think it should be possible to remove the encryption by contacting Bloomberg support. But please note that I'm not related to Bloomberg in any way, so I might be wrong.

If this option doesn't work for you, I may consider to have the package supporting encrypted files. But I'm afraid I need help from someone at bloomberg for that. So it would be rather complicated I guess

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/eguidotti/RblDataLicense/issues/3#issuecomment-664557011, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQAHEPV5MBY3QBFEDMQEGYDR5W74JANCNFSM4OCTMY4A.

--------------------------------------------------------- Disclaimer --------------------------------------------------------- Important Notice: This e-mail is subject to the e-mail disclaimer of the South African Reserve Bank, which can be viewed at: http://www.resbank.co.za/Disclaimer/Pages/SARB-Disclaimer.aspx Should you be unable to access the link provided, kindly send an email to BSTD-ICT-ServiceDesk@resbank.co.za --------------------------------------------------------- Disclaimer ---------------------------------------------------------

eguidotti commented 4 years ago

Hello @Vannies, I think you are right. The encryption is another problem that you would face after downloading the file, but I'm afraid it will not fix the download itself.

Can you please try the following and let me know if some of them work for you?

# check if libcurl is supported. Should return TRUE
capabilities("libcurl")

# replace with your filename
file <- "Rbl_xxxxx.out.gz"

# url
url  <- RblUrl()
tmp  <- tempfile()

# test 1
utils::download.file(paste(url, file, sep='/'), destfile = tmp)
# test 2
utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "internal")
# test 3
utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "wininet")
# test 4
utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "libcurl")
# test 5
utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "wget")
# test 6
utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "curl")
# test 7
utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "auto")
# test 8
getURL(paste(url, file, sep='/'))
Vannies commented 4 years ago

Good day Emanuele

The results of your requests:

capabilities("libcurl") libcurl TRUE

test 1

utils::download.file(paste(url, file, sep='/'), destfile = tmp) Error in utils::download.file(paste(url, file, sep = "/"), destfile = tmp) : scheme not supported in URL 'sftp://dlxxxxxx:zzzzzzzzzzzzzzzz@dlsftp.bloomberg.com:30206/Rbl_0728104720_1.out.gz'

test 2

utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "internal") Error in utils::download.file(paste(url, file, sep = "/"), destfile = tmp, : scheme not supported in URL 'sftp:// dlxxxxxx:zzzzzzzzzzzzzzzz @dlsftp.bloomberg.com:30206/Rbl_0728104720_1.out.gz'

test 3

utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "wininet") Error in utils::download.file(paste(url, file, sep = "/"), destfile = tmp, : scheme not supported in URL 'sftp:// dlxxxxxx:zzzzzzzzzzzzzzzz @dlsftp.bloomberg.com:30206/Rbl_0728104720_1.out.gz'

test 4

utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "libcurl") trying URL 'sftp:// dlxxxxxx:zzzzzzzzzzzzzzzz @dlsftp.bloomberg.com:30206/Rbl_0728104720_1.out.gz' Content length 404 bytes

test 5

utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "wget") Error in utils::download.file(paste(url, file, sep = "/"), destfile = tmp, : 'wget' call had nonzero exit status

test 6

utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "curl") % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 curl: (60) SSL peer certificate or SSH remote key was not OK More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. Error in utils::download.file(paste(url, file, sep = "/"), destfile = tmp, : 'curl' call had nonzero exit status

test 7

utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "auto") Error in utils::download.file(paste(url, file, sep = "/"), destfile = tmp, : scheme not supported in URL 'sftp:// dlxxxxxx:zzzzzzzzzzzzzzzz @dlsftp.bloomberg.com:30206/Rbl_0728104720_1.out.gz'

test 8

getURL(paste(url, file, sep='/')) [1] "begin 600 Rbl_07281047201.out.gz\nMA<@6[C4K9C@]LR '-?,\"# /[&7:QHI*VPZ9O&Y<Z\[O[D8O%G5S=(@KM@& C\nMI\\"Z1Q76I,>DFB&MEC:!\]U-6?WA$C( $7-6O<=>[, ZF^+4\"G.9)U9C4I\nM\"M=0W^#SURP84K8+O1LS0*2WI76$H'0:\'!.QR :^GBS,T[]#X4ZRK'-UX.\nMJT6US,HIR+G(.^>S]^G=G5I\,>KWAU$_%Q=OV]BO\"VC[2DS%A]X$E90!OJH\nMNC-AAU?54>E+ZG^'IG[J,$S1BH)-D#V>G]4.'D+[(R\[U]:@\\\\WD^[8CT\nG-!]O7^W/YHBDH94T&,<02@E8!P,ZRM+4IU@'Z\"]XS&<+V?@\"B2\n \nend\n"

Regards Johann van der Merwe

From: Emanuele Guidotti [mailto:notifications@github.com] Sent: 28 July 2020 05:50 PM To: eguidotti/RblDataLicense RblDataLicense@noreply.github.com Cc: Johann VanderMerwe johann.vandermerwe@resbank.co.za; Mention mention@noreply.github.com Subject: Re: [eguidotti/RblDataLicense] Unable to download Bloomberg file (#3)

Hello @Vannieshttps://github.com/Vannies, I think you are right. The encryption is another problem that you would face after downloading the file, but I'm afraid it will not fix the download itself.

Can you please try the following and let me know if some of them work for you?

check if libcurl is supported. Should return TRUE

capabilities("libcurl")

replace with your filename

file <- "Rbl_xxxxx.out.gz"

url

url <- RblUrl()

tmp <- tempfile()

test 1

utils::download.file(paste(url, file, sep='/'), destfile = tmp)

test 2

utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "internal")

test 3

utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "wininet")

test 4

utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "libcurl")

test 5

utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "wget")

test 6

utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "curl")

test 7

utils::download.file(paste(url, file, sep='/'), destfile = tmp, method = "auto")

test 8

getURL(paste(url, file, sep='/'))

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/eguidotti/RblDataLicense/issues/3#issuecomment-665120375, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQAHEPSCYYERXLDZGM523H3R53XSVANCNFSM4OCTMY4A.

--------------------------------------------------------- Disclaimer --------------------------------------------------------- Important Notice: This e-mail is subject to the e-mail disclaimer of the South African Reserve Bank, which can be viewed at: http://www.resbank.co.za/Disclaimer/Pages/SARB-Disclaimer.aspx Should you be unable to access the link provided, kindly send an email to BSTD-ICT-ServiceDesk@resbank.co.za --------------------------------------------------------- Disclaimer ---------------------------------------------------------

eguidotti commented 4 years ago

Great! It seems we spot the bug. The download.file file function uses a different default method on Windows that does not support SFTP. This should be fixed now. Please try the following:

# install the (hopefully) fixed version 
remotes::install_github("eguidotti/RblDataLicense")

# replace with your filename
file <- "Rbl_xxxxx.out.gz"

# download
localfile <- RblDownload(file)

# read the local file
readLines(localfile)

You should be able to download and read the encrypted file now. As soon as the encryption is removed, you should be able to parse the file as well and succesfully complete the All in One example. Please keep me posted, so I can update the package on CRAN for the other users. Many thanks!

Vannies commented 4 years ago

Hi Emanuele

THANK YOU, I’m able to download the file although it is still in the encrypted format.

I have spoken to the Bloomberg support desk and they indicated that if a file is encrypted it will have an .enc extension. My network guys say that because the connection is to an sftp server the data movement in the connection will be encrypted. I have also read in a Bloomberg document (https://data.bloomberglp.com/professional/sites/10/Bloomberg-SFTP-Connectivity-SSH-Host-Keys.pdf) that they use the SSH protocol version 2.0 encryption on their sftp connection. Since the files do not have the .enc extension my deduction is that the encryption issue may be the SSH protocol. Do you know if RCurl do SSH authentication when the connection is made and the file is downloaded from Bloomberg? I would presume that if this authentication is not done the SSH encryption will not happen correctly.

Regards Johann van der Merwe

From: Emanuele Guidotti [mailto:notifications@github.com] Sent: 29 July 2020 08:07 PM To: eguidotti/RblDataLicense RblDataLicense@noreply.github.com Cc: Johann VanderMerwe johann.vandermerwe@resbank.co.za; Mention mention@noreply.github.com Subject: Re: [eguidotti/RblDataLicense] Unable to download Bloomberg file (#3)

Great! It seems we spot the bug. The download.file file function uses a different default method on Windows that does not support SFTP. This should be fixed now. Please try the following:

install the (hopefully) fixed version

remotes::install_github("eguidotti/RblDataLicense")

replace with your filename

file <- "Rbl_xxxxx.out.gz"

download

localfile <- RblDownload(file)

read the local file

readLines(localfile)

You should be able to download and read the encrypted file now. As soon as the encryption is removed, you should be able to parse the file as well and succesfully complete the All in One example. Please keep me posted, so I can update the package on CRAN for the other users. Many thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/eguidotti/RblDataLicense/issues/3#issuecomment-665816805, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQAHEPS7B3KIK2GOIV6C4MDR6BQNXANCNFSM4OCTMY4A.

--------------------------------------------------------- Disclaimer --------------------------------------------------------- Important Notice: This e-mail is subject to the e-mail disclaimer of the South African Reserve Bank, which can be viewed at: http://www.resbank.co.za/Disclaimer/Pages/SARB-Disclaimer.aspx Should you be unable to access the link provided, kindly send an email to BSTD-ICT-ServiceDesk@resbank.co.za --------------------------------------------------------- Disclaimer ---------------------------------------------------------

eguidotti commented 4 years ago

Hello @Vannies, glad it works.

Regarding the encryption, I know the package works smoothly on Unix like systems. It is possible that there are still some differences due to Windows.

To check this out I'd suggest the following test: download the file in some other way not involving R (e.g. directly from bloomberg website or software). Then unzip and open the file in a text editor. If the file happens to be still encrypted, bloomberg should be able to help you out. Otherwise please let me know and we can try to figure out if this is due to the ssh connection or something else.

Vannies commented 4 years ago

Hi Emanuele

I did two downloads with Filezilla. I could not unzip the zipped file, however, using Notepad both files showed gibberish. I have spoken to Bloomberg but the answer was not clear. I will try again.

Is it possible to check with the author of issue #2 what he did to resolve the matter? I checked but could not see an e-mail adres.

I would like to test something. If I want to change the url to the sftp server, how can I do it?

Regards Johann van der Merwe

From: Emanuele Guidotti [mailto:notifications@github.com] Sent: 30 July 2020 06:44 PM To: eguidotti/RblDataLicense RblDataLicense@noreply.github.com Cc: Johann VanderMerwe johann.vandermerwe@resbank.co.za; Mention mention@noreply.github.com Subject: Re: [eguidotti/RblDataLicense] Unable to download Bloomberg file (#3)

Hello @Vannieshttps://github.com/Vannies, glad it works.

Regarding the encryption, I know the package works smoothly on Unix like systems. It is possible that there are still some differences due to Windows.

To check this out I'd suggest the following test: download the file in some other way not involving R (e.g. directly from bloomberg website or software). Then unzip and open the file in a text editor. If the file happens to be still encrypted, bloomberg should be able to help you out. Otherwise please let me know and we can try to figure out if this is due to the ssh connection or something else.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/eguidotti/RblDataLicense/issues/3#issuecomment-666516091, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQAHEPWUG6TCEBPP7XW6HTLR6GPL3ANCNFSM4OCTMY4A.

--------------------------------------------------------- Disclaimer --------------------------------------------------------- Important Notice: This e-mail is subject to the e-mail disclaimer of the South African Reserve Bank, which can be viewed at: http://www.resbank.co.za/Disclaimer/Pages/SARB-Disclaimer.aspx Should you be unable to access the link provided, kindly send an email to BSTD-ICT-ServiceDesk@resbank.co.za --------------------------------------------------------- Disclaimer ---------------------------------------------------------

eguidotti commented 4 years ago

Hello @Vannies

I did two downloads with Filezilla. I could not unzip the zipped file, however, using Notepad both files showed gibberish. I have spoken to Bloomberg but the answer was not clear. I will try again.

Then yes, I'm afraid it's an issue on the bloomberg side. Or at least, this is not due to the ssh transfer.

Is it possible to check with the author of issue #2 what he did to resolve the matter? I checked but could not see an e-mail adres.

Unfortunately I have no more information (nor contact) than what publicly available here on github.

I would like to test something. If I want to change the url to the sftp server, how can I do it?

Sure, please have a look at the RblDataLicense documentation. In particular you can use the arguments of the RblConnect function. 

RblConnect(user, pw, host = "dlsftp.bloomberg.com", port = "30206", protocol = "sftp", verbose = TRUE)

Keep me posted on your progress, I'll be waiting to close this issue before updating the package on CRAN. Thanks!

Vannies commented 4 years ago

Hi Emanuele

Thank you for the reply. I will keep you posted.

My url change did not work for now.

I ran the all in one example for the last time today and it gave the following results:

data <- RblQuery(fields = c('PX_LAST'),

  • identifiers = c('WZ1 Comdty'),
  • from = '2020-07-25',
  • to = '2020-07-28') Uploading file Rbl_0731143216_1.req Successfully uploaded file Rbl_0731143216_1.req Checking if file Rbl_0731143216_1.out.gz is available... trying URL 'sftp://dxxxxxx:zzzzzzzzzzzzzzzz@dlsftp.bloomberg.com:30206/Rbl_0731143216_1.out.gz' File not yet available, waiting... .............................. Checking if file Rbl_0731143216_1.out.gz is available... trying URL 'sftp:// dxxxxxx:zzzzzzzzzzzzzzzz @dlsftp.bloomberg.com:30206/Rbl_0731143216_1.out.gz' Content length 418 bytes File available! Downloading... Error in if (programname == "gethistory") { : argument is of length zero

Then I did the following and the file was downloaded:

localfile <- RblDownload(file) Checking if file Rbl_0731143216_1.out.gz is available... trying URL 'sftp:// dxxxxxx:zzzzzzzzzzzzzzzz @dlsftp.bloomberg.com:30206/Rbl_0731143216_1.out.gz' Content length 418 bytes File available! Downloading...

read the local file

readLines(localfile) [1] "begin 600 Rbl_07311432161.out.gz" [2] "MA<@6[C4K9C@3P^F^GIP0Y8YO&S\"V5V;MF@S5'5N^]21XHYP!#[8\";],M" [3] "MDV'2,CVDY;Q!YR,\"/Q@+^[VJT$N+'S^1Q#QM;9N#!00'4.-!5#!V++^M-6=" [4] "M!BL8QEC75-K4X%DGJ#3R$KIQ[$AM@C+)8=E6RY^EG>:+J<+.+F>NCST!ZD" [5] "MF:)T;>2,_WVA?S+-B^V)^]7DV9RPP@Z<GI4MPB^D4TKME5\"T4J?T-CICXZE" [6] "M<-O7)NY;/I+:0^V]\"-1[XX>E79R?!$=QC<4+E8VR^: !0NT15JW5;8=NSTF" [7] "M2GF\SD5S3LJESV4I9A:-VZG,>'IZ2(%-7;'$%/U5,5 _]*(U=#@R5Q[=+Q" [8] "\"TX, " [9] " " [10] "end"

Is something wrong?

Regards Johann van der Merwe Data Steward Economic Research Department

P O Box 427 Pretoria 0001 South Africa 370 Helen Joseph Street (formerly Church Street) Pretoria 0002 Tel. +27 12 313 3940 Cell 082 452 9999 E-mail: johann.vandermerwe@resbank.co.zamailto:johann.vandermerwe@resbank.co.za www.resbank.co.zahttp://www.resbank.co.za/

[cid:image002.png@01D66750.36411D90]http://www.resbank.co.za/

From: Emanuele Guidotti [mailto:notifications@github.com] Sent: 29 July 2020 08:07 PM To: eguidotti/RblDataLicense RblDataLicense@noreply.github.com Cc: Johann VanderMerwe johann.vandermerwe@resbank.co.za; Mention mention@noreply.github.com Subject: Re: [eguidotti/RblDataLicense] Unable to download Bloomberg file (#3)

Great! It seems we spot the bug. The download.file file function uses a different default method on Windows that does not support SFTP. This should be fixed now. Please try the following:

install the (hopefully) fixed version

remotes::install_github("eguidotti/RblDataLicense")

replace with your filename

file <- "Rbl_xxxxx.out.gz"

download

localfile <- RblDownload(file)

read the local file

readLines(localfile)

You should be able to download and read the encrypted file now. As soon as the encryption is removed, you should be able to parse the file as well and succesfully complete the All in One example. Please keep me posted, so I can update the package on CRAN for the other users. Many thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/eguidotti/RblDataLicense/issues/3#issuecomment-665816805, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQAHEPS7B3KIK2GOIV6C4MDR6BQNXANCNFSM4OCTMY4A.

--------------------------------------------------------- Disclaimer --------------------------------------------------------- Important Notice: This e-mail is subject to the e-mail disclaimer of the South African Reserve Bank, which can be viewed at: http://www.resbank.co.za/Disclaimer/Pages/SARB-Disclaimer.aspx Should you be unable to access the link provided, kindly send an email to BSTD-ICT-ServiceDesk@resbank.co.za --------------------------------------------------------- Disclaimer ---------------------------------------------------------

eguidotti commented 4 years ago

From what I see, the RblQuery is able to generate the request file, upload the request file, download the response file, but it is not able to parse the response file. I see RblDownload works now, but if you try RblParse it will fail. RblParse is the last step of the RblQuery and it needs a properly formatted file. You should be able to open the file in a text editor and read the values, otherwise

In both cases the RblParse function will not be able to parse the file. I'd suggest to ask support until you can download the file (eg Filezilla) and open in a text editor in a human readable format. Once you are able to obtain a human readable file in some way, I guess it will be quick to fix this in the package.

eguidotti commented 4 years ago

Hello @Vannies, can I ask you if you have some news on the issue? Many thanks, Emanuele

Vannies commented 4 years ago

Hi Emanuele

At long last I was able to decrypt the Bloomberg file. I am able to submit and download the file from Bloomberg then I have to run the decryption process after which I can use the RblParse function. As stated in your reply I am not able to use the all in one process however using the step by step process I am able to do the download. Thank you for your help!!

PS. Maybe you should consider to make the decryption part of the package. I think it might not be to difficult. Speak to Bloomberg helpdesk they should be able to assist you.

Kind regards Johann van der Merwe Data Steward Economic Research Department

P O Box 427 Pretoria 0001 South Africa 370 Helen Joseph Street (formerly Church Street) Pretoria 0002 Tel. +27 12 313 3940 Cell 082 452 9999 E-mail: johann.vandermerwe@resbank.co.zamailto:johann.vandermerwe@resbank.co.za www.resbank.co.zahttp://www.resbank.co.za/

[cid:image003.png@01D68B56.0D1A8B20]http://www.resbank.co.za/

From: Emanuele Guidotti [mailto:notifications@github.com] Sent: 31 July 2020 05:49 PM To: eguidotti/RblDataLicense RblDataLicense@noreply.github.com Cc: Johann VanderMerwe johann.vandermerwe@resbank.co.za; Mention mention@noreply.github.com Subject: Re: [eguidotti/RblDataLicense] Unable to download Bloomberg file (#3)

From what I see, the RblQuery is able to generate the request file, upload the request file, download the response file, but it is not able to parse the response file. I see RblDownload works now, but if you try RblParse it will fail. RblParse is the last step of the RblQuery and it needs a properly formatted file. You should be able to open the file in a text editor and read the values, otherwise

In both cases the RblParse function will not be able to parse the file. I'd suggest to ask support until you can download the file (eg Filezilla) and open in a text editor in a human readable format. Once you are able to obtain a human readable file in some way, I guess it will be quick to fix this in the package.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/eguidotti/RblDataLicense/issues/3#issuecomment-667191856, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQAHEPXUMEGJ4LPYFTYAGGTR6LRWBANCNFSM4OCTMY4A.

--------------------------------------------------------- Disclaimer --------------------------------------------------------- Important Notice: This e-mail is subject to the e-mail disclaimer of the South African Reserve Bank, which can be viewed at: http://www.resbank.co.za/Disclaimer/Pages/SARB-Disclaimer.aspx Should you be unable to access the link provided, kindly send an email to BSTD-ICT-ServiceDesk@resbank.co.za --------------------------------------------------------- Disclaimer ---------------------------------------------------------

eguidotti commented 4 years ago

Hello @Vannies, thank you for your message. I'm happy you managed to get the data. I updated the README, submitted the new version to CRAN and set up this website as a reference for the package. I'm not going to implement the encryption at the moment as it seems to be discouraged by Bloomberg itself. However, I may consider this feature in the future if other people are striving with that. Thanks for improving the package! All the best, Emanuele