Code for libcurl wrappers

[GoogleCodeExporter]

I have written a mostly complete wrapper around libcurl.  It is very bare-metal 
(almost identical 
to the easy_curl interface), however from that, more "javascript-y" like apis 
could be created 
(using javascript).   I think it's "OK" -- my guess it could use some other 
eyes, but at least the 
boiler plate is created.

I also got a partial collection of common posix system calls all wrapped up -- 
so you can 
pretend you are writing in C

var fd = posix.open("/tmp/foobar", posix.O_CREAT | posix.O_TRUNC | 
posix.O_WONLY);

somewhat in the spirit of python's 
http://docs.python.org/library/os.html#module-os

I don't think it would take too much work to convert them into GPSEE format, 
and use the 
module loader/require stuff.    So the million dollar question is do you want 
them?  I'm happy to 
re-assign all rights and use whatever license.

Original issue reported on code.google.com by nickgsup...@gmail.com on 5 Jan 2010 at 5:41

[GoogleCodeExporter]

Hi, Nick!

I would definitely be interested in "seeing the code" - particularly the curl 
module.
Easy-curl functionality would nice to add (and is in fact on my private to-do 
list).

Your module-os code probably has some cross-over against both FFI and fs-base, 
but I
don't see that as a reason to exclude it. There is no real penalty against bloat
here, since we're modularized; only in volume-of-code-to-maintain.

Attaching appropriate license blocks (to match rest of code) is sufficient for
contribution. 

You might want to read this (horribly incomplete) wiki page:
http://code.google.com/p/gpsee/wiki/JSAPI_GPSEE as well.

To summarize, take your curl/module-os code, stuff it in a directory under 
modules,
change jsapi.h to gpsee.h, add the module boiler plate at the bottom and 
compile. I'm
often around on #gpsee on freenode if you get stuck. 

Original comment by wes%page...@gtempaccount.com on 5 Jan 2010 at 6:59

[GoogleCodeExporter]

Ok, rev 1 of gpsee-fied wrappers around libcurl.   To install, untar in the 
module directory.  You should then 
have a curl directory.  Modify the module.mk as desired, and do a top level 
make build/install.

It that directory is *sample* high level http client named 'htpp.js'
    gsr -f 'http.js' 
should fetch http://www.google.com/ and print  the header, body and a few stats.

libcurl is a bit unusual.  I've laid out some the issues / designs / todos  in 
the README.txt file.   It all "works" 
with the one big exception of any http body data should be using a ByteArray.  
Right now it uses a string, and 
has to strip out any byte > 127 to make sure the UTF8 conversion works.  (this 
is a gross hack I know).   I'm 
not sure how to include ByteArray in another C module.

let me know what you think.

Original comment by nickgsup...@gmail.com on 10 Jan 2010 at 10:19

Attachments:

• curl_module.tar.gz

[GoogleCodeExporter]

Message on the mailing list from nickg:

I've attempted to use google code's hg clone hosting.  Excuse the
silly names, I had no idea how all the form fields would be used.

Anyways... the only changes are a new directory "modules/curl".
http://code.google.com/r/nickgsuperstar-libcurl/source/browse/#hg/modules/curl

Notes:

* You might have to edit the modules.mk to point to the libcurl
include files.
* I followed the libcurl API closely... it's not very javascripty or
highlevel.  My
* The http.js is sample high level API..., http + libcurl is
suprisingly complex, but
   grs -f http.js
    should print out some stuff on www.google.com
* The C coding style I used is -wrong-, I'll fix that.
* Memory.h/.c is a  bytething hack that needs some scrubbing, but it
works.
* Pretty comfortable with memleaks but not hard tested,
* Might be a few not-quite-correctly-rooted jsvals here and there --
this code is pretty old actually haven't really audited it.
* there is a "makecurl.py" file which generates some code based on the
curl/curl.h #DEFINES... it's a real pain in the ass, but generates a
switch to determine if a property is a int/double/callback/slist or
other.

Original comment by wes%page...@gtempaccount.com on 16 Jan 2010 at 2:27

• Changed state: Started
• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

My clone is updated.  Eliminates the easycurl_callback class which in turn 
eliminates a 
few bugs and simplifies the api.

Looking at the commonjs.org site, looks like narwhal has a http client / spec.  
Should be 
simple to match that interface if desired.  

Original comment by nickgsup...@gmail.com on 18 Jan 2010 at 1:15

[GoogleCodeExporter]

Hi folks,

The Flusspferd (?) guys have a nice libcurl interface, which is mostly the 
same.  They also use Binary/B (BTW, 
what is actually wrong with B that makes it not acceptable as the standard?).  
The odds of a standard 
HTTPClient interface (other have client XHR) is unlikely anytime soon.

I'm going to talk to them and see about duplicating the interface in gysee.  
This way any future JS code to 
make a true HTTP client should identical on both platforms.

Suggestions?

--nickg

Original comment by nickgsup...@gmail.com on 13 Mar 2010 at 6:44

[GoogleCodeExporter]

Sorry I haven't had the chance to review this yet; I have been absolutely 
*swamped*.

Synchronizing APIs with Flusspferd is a fantastic goal. I have made several 
changes
to gpsee and gsr specifically to help our respective products run the same JS 
code.

Binary/B is not a bad standard, but it's not great.   It has some specification
sloppiness, and there's a lot there.  If we ever move away from binary/b in 
GPSEE, we
will still support require('binary-b') or similar.

As for a standard HTTPClient interface -- synchronous XHR could be layered on 
top of
curl, and asynchronous is doable down-the-road... when we add an event loop. I 
see
XHR as the most likely standard interface in the short to medium term.

Original comment by wes%page...@gtempaccount.com on 13 Mar 2010 at 7:05

[GoogleCodeExporter]

Ok, I'll poke around libcurl with flusspferd guys.

I've written an synchronous XHR-in-javascript with libcurl already using my 
before-i-knew-about-gpsee 
spidermonkey-hack.  Once I get libcurls in sync, I'll make that and the charset 
sniffer public.

two gotchas with XHR:
* not designed to 'just give me the raw data'
* errors are dependent on DOMErrors which may or may not make sense.

both easy enough to fix.   No rush on async.

Original comment by nickgsup...@gmail.com on 13 Mar 2010 at 7:17

[GoogleCodeExporter]

Thanks, Nick.

FWIW -- I would totally take an XHR module that was built on require("curl") 
into the
GPSEE standard library.

Wes

Original comment by wes%page...@gtempaccount.com on 13 Mar 2010 at 9:17

[GoogleCodeExporter]

XHR: cool, I can add that to my branch quite soon.  The first version will be 
missing the charset sniffer, but so 
far every version I have ever seen in the wild seems to missing that as well.

Speaking of which, I just refreshed/remerged the "nickgsuperstar-libcurl' 
branch.  Seems to compile and 
recognize the module (after a few renames and other minor changes).

The flusspferd seem ok with me sync-ing APIs as well.

onward!

Original comment by nickgsup...@gmail.com on 14 Mar 2010 at 7:21

[GoogleCodeExporter]

update: flusspferd api is a touch more complicated and  works a bit 
differently.  Mime is very "it's just like C -
- use libcurl doco", while theres is a bit more "javascripty" and a bit more 
high level.  With some more work I I 
think a compatibility module could be done (all in javascript) if desired.   
The main difference is that their 
binary/b module is just part of the raw api so it's easy to use in other C 
modules, while in gpsee that is more 
tricky.

Instead I focused on making a XHR module which is checked in.    Still a bit 
rough but works.  the test.js in the 
modues directory

Original comment by nickgsup...@gmail.com on 15 Mar 2010 at 4:04

[GoogleCodeExporter]

I'm reviewing this right now!

Original comment by donny.vi...@gmail.com on 26 Mar 2010 at 2:50

[GoogleCodeExporter]

This isn't a gzip file!

Original comment by donny.vi...@gmail.com on 26 Mar 2010 at 2:51

[GoogleCodeExporter]

not a gzip file?  It's a branch/clone or whatever mercurial calls it.  Look at 
the clones tab/link(?) on google code.
You'll see something like 'nickgsuperstar-gpsee-curl' or something.  Look in 
modules/curl modules/xhr.js

send me a note if you need help.

--nickg

Original comment by nickgsup...@gmail.com on 26 Mar 2010 at 3:35

[GoogleCodeExporter]

nickg: hey i figured it out, it was a bzip2 archive even though it had a tar.gz
extension :)

i'm having a problem building though. you should add a rule to module.mk to make
libcurl_curlinfo.h (that is the file that gets created by makecurl.py, right?)

also i think i might need to know what version of libcurl you're using because 
i seem
to be missing CURLOPT_SSH_KNOWNHOSTS, CURLOPT_SSHKEYDATA and 
CURLOPT_SSH_KEYFUNCTION.
I installed the OpenSSL variety (vis a vis GNU TLS) so maybe that's the problem?
Please let me know, and thank you for contributing to GPSEE! :)

Original comment by donny.vi...@gmail.com on 26 Mar 2010 at 6:59

[GoogleCodeExporter]

Oh crap... Donny.. that tarball is ancient.  The GOOD stuff is an HG clone 
hosted on googlecode and it's been 
recently synced up.

Pull down this instead:
http://code.google.com/r/nickgsuperstar-libcurl/source/browse

Original comment by nickgsup...@gmail.com on 26 Mar 2010 at 7:03

[GoogleCodeExporter]

Cool, thanks, got it :)

Is there a way I can chat with you?

Original comment by donny.vi...@gmail.com on 26 Mar 2010 at 7:17

[GoogleCodeExporter]

donny's notes:
http://pastebin.mozilla.org/710767

Original comment by nickgsup...@gmail.com on 26 Mar 2010 at 8:51

[GoogleCodeExporter]

http://code.google.com/r/nickgsuperstar-libcurl/source/browse/modules/curl/curl.
c?
r=df90cc11a1ae282ded421b7461ba52892d2bf8f9

has most of donny's revisions.  The ones that are missing are:

JS_Suspend/Resume calls.  I have them in commented out in most of the places.  
Do I need to capture and 
save the refcount in my private data?

Some other item on when to set the context.  I'm still reading.

There are a bunch of "TODO" item marked in the code.

Testing: 

#  does a little bit of everything
gsr -d -f modules/curl/easytest.js

# tests the xhr module
gsr -d -f modules/xhr/test.js

Original comment by nickgsup...@gmail.com on 28 Mar 2010 at 9:14

[GoogleCodeExporter]

Original comment by nickgsup...@gmail.com on 2 Apr 2010 at 12:02

[GoogleCodeExporter]

merged with mainline and cleaned up.   Added debug callback as well.

Not even sure how it worked before (but it did), but this should be solid.  
Tested with lasted firefox alpha under 
debug mode.

Original comment by nickgsup...@gmail.com on 29 Apr 2010 at 12:08

[GoogleCodeExporter]

Nick, I merged your clone in with mainline GPSEE.

Original comment by wes%page...@gtempaccount.com on 21 Jun 2010 at 8:50

• Changed state: Fixed

[GoogleCodeExporter]

Holy smokes!

I'll be "back on the job" shortly to take a look!

Original comment by nickgsup...@gmail.com on 22 Jun 2010 at 2:45