search

gnudatalanguage / gdl

GDL - GNU Data Language
GNU General Public License v2.0
277 stars 61 forks source link

Anybody ever get wget to work? #513

Closed cosmicbrucee closed 12 months ago

cosmicbrucee commented 6 years ago

Dear gdl users/developers: wget is an astro library function to request an http get (or post).

I cannot get it to work, nor get an informative error message: GDL> a=webget('http://www.yahoo.com') % Compiled module: WEBGET. % SOCKET: WEBGET: Unable to lookup host. Unit: 100, File: % Execution halted at: WEBGET 154 /Users/Bruce/idlpro/gsfc-astro/pro/webget.pro % $MAIN$
GDL>

This has the flavor to me of some port needing to be open, and having nothing to do with gdl itself.

Any help in getting this working would be much appreciated.

Thanks,

-Bruce

alaingdl commented 6 years ago

The example you are using is not working in IDL too. With 

a=webget('http://www.mpia.de/index.html')

I received the same output in IDL and GDL (current git)

alaingdl commented 6 years ago

second example in file webget.pro is not working in IDL or GDL too

alaingdl commented 6 years ago

just to mention : webget.pro is based on the use of SOCKET. On my laptop, the IDL examples on SOCKET are not working. Alternatively, you can use : 

QueryDSS, '00.00+30.00', image, header, survey = '2b'

in the AstroLib.

cosmicbrucee commented 6 years ago

Salut Alain!

QueryDSS will not help me - The point is that I want to use gdl for HTTP communications - in fact, there are lots of devices that communicate via an HTTP server.

I suppose I could just use a system escape, i.e. spawn, to the system’s HTTPget command, but I dont’ know how to do that, I’ll have to learn….

I will also post this to the IDL group.

Thanks,

-Bruce

On Nov 9, 2018, at 2:53 AM, Alain notifications@github.com wrote:

just to mention : webget.pro is based on the use of SOCKET. On my laptop, the IDL examples on SOCKET are not working. Alternatively, you can use :

QueryDSS, '00.00+30.00', image, header, survey = '2b' in the AstroLib.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/gnudatalanguage/gdl/issues/513#issuecomment-437324151, or mute the thread https://github.com/notifications/unsubscribe-auth/Aqz1292eIpWMogHjsYcGGlGj3_JItwTTks5utV6MgaJpZM4YWCtS.

slayoo commented 6 years ago 
GDL> spawn, 'curl http://www.mpia.de/index.html', a

?

alaingdl commented 6 years ago

Yes, @slayoo but the devil in details : I did not succeed to call the example (provided by WEBGET())

wget 'archive.eso.org/dss/dss/image?ra=0.00000&dec=30.0000&x=10&y=10&Sky-Survey=DSS1&mime-type=download-fits'

from SPAWN in IDL or GDL, but this is working perfectly at (direct bash) command line level

@cosmicbrucee could you provide a template of what you would like to do ?

cosmicbrucee commented 6 years ago

Dear Sylwester:

I have no problem using curl through spawn in gdl on a linux system:

GDL> spawn,'curl -L www.yahoo.com',result % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 8 100 8 0 0 41 0 --:--:-- --:--:-- --:--:-- 51 100 485k 0 485k 0 0 319k 0 --:--:-- 0:00:01 --:--:-- 406k GDL> help,result RESULT STRING = Array[3897] GDL>

My main question / point here is that when you escape to the system, instead of using an IDL function, you lose portability of code, a big advantage of idl/gdl. For example, how do you do a curl command on a windows machine? I have no idea about using curl on windows machines, and have none to test here now. Unfortunately, various instruments may require me to do so.

-Bruce

On Nov 9, 2018, at 2:33 PM, Sylwester Arabas notifications@github.com wrote:

GDL> spawn, 'curl http://www.mpia.de/index.html', a ?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/gnudatalanguage/gdl/issues/513#issuecomment-437515970, or mute the thread https://github.com/notifications/unsubscribe-auth/Aqz12-DcYb2QNZSdJg9a3GfPzJHrydsnks5utgKygaJpZM4YWCtS.

cosmicbrucee commented 6 years ago

Re: your question of: “what I would like to do”: I need to both do a standard “get” of information as well as posting some arguments to the HTTP request.

In my last post to this group, I showed that you could use spawn with linux curl utility to do an http get: IDL>spawn,’curl -L www.yahoo.com http://www.yahoo.com/',result

I also noted that not going with an IDL/gdl command you loose machine independence. Specifically, I have no idea if curl works on a windows machine.

Continuing with the linux machine, however, I am pretty sure I have a solution when you have to do a post operation, i.e. when you have to send arguments over http: I haven’t tested this yet, but I believe this will work fine:

spawn,’curl -d “device=mount&cmd=move&ra2000=11 22 33&dec2000=88 54 32” http://127.0.0.1 http://127.0.0.1:8080/', result

where “device=mount&cmd=move&ra2000=11 22 33&dec2000=88 54 32” is the argument passed in the http request. This should be equivalent to typing into a browser : http://127.0.0.1 http://127.0.0.1:8080//?device=mount&cmd=move&ra2000=11 22 33&dec2000=88 54 32 … i.e. a command of the format: [URL]?[arguments].

The only question for a linux system then, I think, is the quotation marks - I may need to escape the inner double quotes or something.

Again, no idea if this will work in Windows, or if I could install curl on a windows machine.

-B

On Nov 10, 2018, at 1:44 AM, Alain notifications@github.com wrote:

Yes, @slayoo https://github.com/slayoo but the devil in details : I did not succeed to call the example (provided by WEBGET())

wget 'archive.eso.org/dss/dss/image?ra=0.00000&dec=30.0000&x=10&y=10&Sky-Survey=DSS1&mime-type=download-fits' from SPAWN in IDL or GDL, but this is working perfectly at (direct bash) command line level

@cosmicbrucee https://github.com/cosmicbrucee could you provide a template of what you would like to do ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gnudatalanguage/gdl/issues/513#issuecomment-437571715, or mute the thread https://github.com/notifications/unsubscribe-auth/Aqz123n4gqO2ytkKpabILRLZAQH3Uasrks5utqAEgaJpZM4YWCtS.

slayoo commented 6 years ago

@alaingdl this seems to work (and also perhaps solve the escape/argument passing issue for @cosmicbrucee ):

GDL> spawn, /noshell, ['curl', 'http://archive.eso.org/dss/dss/image?ra=0.00000&dec=30.0000&x=10&y=10&Sky-Survey=DSS1&mime-type=download-fits'], a
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  264k  100  264k    0     0   303k      0 --:--:-- --:--:-- --:--:--  303k
GDL> help, a
A               STRING    = Array[748]
cosmicbrucee commented 6 years ago

This is getting very positive: On the IDL google group

https://groups.google.com/forum/#!topic/idl-pvwave/AsL0qbCZ_Z8

people answered that webget works fine after version 8. Indeed, my machine with version 8 works fine, and has a different version of webget. My IDL machine v7.1.1. does NOT work.

So, what can we do to get this working in gdl?

Let me reproduce my reply here:

Thank you very much to all who answered. Indeed, something was fixed somewhere between 7.1.1 and 8.5.1, where it works fine.

So, what would it take to get it working on my older versions of IDL and um, I hope I can mention this here, gdl. Can I just get newer version of the code?

It would be really helpful to those of us with limited licenses and students and etc. if we could help get this running on gdl, so I would much appreciate anyone with suggestions after looking at the following:

my IDL 8.5.1 machine version says "V5.4 (uses SOCKET)... then the last comment is "Barth, Nov. 2006"

My IDL 7.1.1 machine versions says "V5.4 (uses Socket).... but then the last comment is "Landsman April 2003"

So, I replaced and re-complied the code, and got something that looks like SOCKET is trying to look at only local sockets, or maybe it's port is blocked:

IDL> .run ~/idlpro/gsfc-astro/pro/webget.pro % Compiled module: MIMETYPE. % Compiled module: WEBGET. IDL> aa=webget('yahoo.com')
% SOCKET: Unable to connect to host. Unit: 100 File: localhost.80 Connection refused % Execution halted at: WEBGET 181 /home/bruce/idlpro/gsfc-astro/pro/webget.pro % $MAIN$
IDL>

So, maybe SOCKET is the culprit? Maybe if we can fix that, and use the v8+ version of webget, we could get this working.

Any ideas on the problem with SOCKET?

Thanks,

-B

On Nov 11, 2018, at 11:37 PM, Sylwester Arabas notifications@github.com wrote:

@alaingdl this seems to work (and also perhaps sove the escape/argument passing issue for @cosmicbrucee ):

GDL> spawn, /noshell, ['curl', 'http://archive.eso.org/dss/dss/image?ra=0.00000&dec=30.0000&x=10&y=10&Sky-Survey=DSS1&mime-type=download-fits' ], a % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload
Total Spent Left Speed

100 264k 100 264k 0 0 303k 0 --:--:-- --:--:-- --:--:-- 303k GDL

help , a A
STRING = Array[748] — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

UC Berkeley Space Sciences Laboratory Bruce_Grossan@lbl.gov voice- 510.735.8806 fax - please send scans by email

cosmicbrucee commented 6 years ago

There are very interesting developments on webget on the IDL message board, so I'd like to share there here:

Originally we found that webget() will not get a URL unless you are in IDL v8+

Next, "Lajos Foldy" noted that the code requires a trailing slash in the URL; it turns out that helps a lot....

" IDL7.1.1 - GENERALLY GOOD NEWS

(Note that redirects can be handled by IDL>spawn,'curl -L [url]',result ) "

...

"SAME MACHINE, gdl 0.9.5:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

404 Not Found

Not Found

The requested URL /~bruce/ was not found on this server.

Apache/2.2.3 (Scientific Linux) Server at www-cdf.lbl.gov Port 80
0} GDL> " So, is there some difference in socket then? At any rate, it is not giving the same request in the same way. Any ideas would be much appreciated. -B
GillesDuvert commented 4 years ago

works for me. Try a=webget('http://www.google.com/')

GillesDuvert commented 3 years ago

not sure this works?

cosmicbrucee commented 3 years ago

This does not work for me either. At times I have had limited success adding a trailing slash. At this point, I use spawn,'curl -L [url]' for the best results.

B

GDL> result=webget('http://google.com') % SOCKET: WEBGET: Unable to lookup host. Unit: 103, File: % Execution halted at: WEBGET 154 /Users/bruce/idlpro/gsfc-astro/pro/webget.pro % WEBGET 154 /Users/bruce/idlpro/gsfc-astro/pro/webget.pro % WEBGET 154 /Users/bruce/idlpro/gsfc-astro/pro/webget.pro % WEBGET 154 /Users/bruce/idlpro/gsfc-astro/pro/webget.pro % $MAIN$
GDL>

On Mar 19, 2021, at 12:51 PM, Giloo @.***> wrote:

not sure this works?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

UC Berkeley Space Sciences Laboratory @.*** voice- 510.735.8806 fax - please send scans by email

GillesDuvert commented 3 years ago

in fact, works at times.

GDL> a=webget('http://www.google.com/')
% Compiled module: WEBGET.
GDL> a
{ HTTP/1.1 200 OK Date: Sun, 21 Mar 2021 18:37:34 GMT Expires: -1 Cache-Control: private, max-age=0 Content-Type: text/html; charset=ISO-8859-1 (...)
GillesDuvert commented 12 months ago

closed, webget works, it's just the other end that does not work, as "http" is now refused in favor or 'https".