search

scripting / drummerRFC

A place to post RFCs for people who use and develop in Drummer.
MIT License
11 stars 0 forks source link

New verb: http.client #6

Open scripting opened 2 years ago

scripting commented 2 years ago

Here's the DocServer page for the new verb.

scripting commented 2 years ago

Good morning! ;-)

I had a few loose-ends to tie off last night when I added the change note about the http.client verb.

Since then, I've --

  1. Fixed a problem in the implementation of the verb when we are not going through the proxy server.

  2. Created a test app to call from scripts, to verify that POST requests work in at least a rudimentary way.

First experiment

I'd like you to try it out, by adding a command to your Scripts menu. Call it Test HTTP Post.

var theRequest = {
    type: "POST",
    url: "http://postreceiver.scripting.com/uppercase",
    data: "oh give me a home, where the buffalo roam"
    };
console.log (http.client (theRequest))

You should see this in the console.

OH GIVE ME A HOME, WHERE THE BUFFALO ROAM

If you ran the script and it worked, please give a thumbs-up to this comment. If you want to share your thoughts, please leave a comment. This is very fluid right now, if you see things that are wrong, now is the time to speak up.

The postReceiver app

On the other end is an app that handles the post request. It converts the body to uppercase and returns the result.

Here is the source for that app. I have it deployed at postreceiver.scripting.com.

scotthansonde commented 2 years ago

Tried it, yes, it works 👍

On Sat, Nov 6, 2021 at 4:38 PM Dave Winer @.***> wrote:

Good morning! ;-)

I had a few loose-ends to tie off last night when I added the change note about the http.client verb.

Since then, I've --

1.

Fixed a problem in the implementation of the verb when we are not going through the proxy server. 2.

Created a test app to call from scripts, to verify that POST requests work in at least a rudimentary way.

First experiment

I'd like you to try it out, by adding a command to your Scripts menu. Call it Test HTTP Post.

var theRequest = { type: "POST", url: "http://postreceiver.scripting.com/uppercase", data: "oh give me a home, where the buffalo roam" }; console.log (http.client (theRequest))

You should see this in the console.

OH GIVE ME A HOME, WHERE THE BUFFALO ROAM

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/scripting/drummerRFC/issues/6#issuecomment-962468957, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCHNHNSWGQTWE4LK3SJKLDUKVKZHANCNFSM5HOJVLPQ .

-- Scott Hanson

Email: @.*** Mobile: +49-171-5529568 Johmsweg 9a, 21266 Jesteburg, Germany

Day Job: http://www.mcdonalds-nordheide.de/

scripting commented 2 years ago

@papascott -- when i saw you were calling the ajax routine yourself, i realized i needed to do this verb sooner than later. so your validation being the first is very appropriate. thanks! ;-)

scotthansonde commented 2 years ago

I take it headers haven't been implemented yet? To trigger a GitHub action I need to set an Authorization header. This is what I am trying: 

var actionURL = "https://api.github.com/repos/papascott/test-github-actions/actions/workflows/14862978/dispatches"
var theRequest = {
    type: "POST",
    url: actionURL,
    data: JSON.stringify({ref: 'main'}),
    headers: {"Authorization": "Bearer "+ root.env.githubToken} 
    };
console.log (http.client (theRequest))

A dialog shows this message: Error running script: Can't read the URL, "https://api.github.com/repos/papascott/test-github-actions/actions/workflows/14862978/dispatches" because we received a status code of 403..

This script using XMLHttpRequest does work from the scripts menu: 👍 

const xhr = new XMLHttpRequest();
var actionURL = "https://api.github.com/repos/papascott/test-github-actions/actions/workflows/14862978/dispatches"
xhr.open("POST", actionURL);
xhr.setRequestHeader('Content-Type', 'application/json');
xhr.setRequestHeader('Authorization', 'Bearer '+ root.env.githubToken);
xhr.send(JSON.stringify({ref: 'main'}));
dialog.alert ('Sent hook to GitHub Actions')
scripting commented 2 years ago

@papascott -- that's correct -- headers have not been implemented yet. today is the day. stay tuned. ;-)

scripting commented 2 years ago

@papascott -- reviewing the code, actually headers should work if you're not using the proxy server.

i haven't checked the server code yet, so it might even work there.

after checking the proxy server code, it clearly won't pass the headers along to the proxy server. so for now you should only expect headers to work if you're not the boolean is false for using the proxy server.

http.client copies everything that's in your options struct into the request, so if you have a headers object that should be copied into the request object.

that said i'm having trouble getting it to work with my test case, so there may be something preventing it from working.

scripting commented 2 years ago

I also added some code to http.client to protect any headers that may be added by default in the future.

Here's the actual code.

client: function (options, flUseProxyServer=true) { //11/5/21 by DW

    var request = { //defaults
        type: "GET",
        url: undefined,
        data: undefined,
        params: undefined,
        headers: new Object ()
        }
    if (options.headers !== undefined) { //11/7/21 by DW
        for (var x in options.headers) {
            request.headers [x] = options.headers [x];
            }
        }
    for (var x in options) {
        if (x != "headers") {
            request [x] = options [x];
            }
        }
    if (request.data !== undefined) {
        if (!$.isPlainObject (request.data) && (typeof (request.data) != "string")) { //8/2/21 by DW
            request.data = request.data.toString ();
            }
        }
    if (request.params !== undefined) {
        request.url += "?" + drummerBuildParamList (request.params);
        }
    if (flUseProxyServer) {
        return new Promise (function (resolve, reject) {
            var proxyRequest = {
                method: request.type,
                url: request.url,
                body: request.data,
                headers: request.headers
                };
            var jsontext = jsonStringify (proxyRequest);
            servercall ("httprequest", {request: jsontext}, true, function (err, data) {
                if (err) {
                    reject (err);
                    }
                else {
                    resolve (data); 
                    }
                });
            });
        }
    else {
        return new Promise (function (resolve, reject) {
            $.ajax (request)
                .success (function (data, status) { 
                    resolve (data); 
                    }) 
                .error (function (status) { 
                    var err = JSON.parse (status.responseText);
                    reject (err);
                    });
            });
        }
    }
scripting commented 2 years ago

@papascott -- I just made a change to the http.client so it's now transmitting the headers to the proxy server, and checked the code in the proxy, and it should just pass it through to the Node.js request function. So at this point I think what's in http.client is correct. I will update the code above to reflect what's actually in Drummer right now.

scotthansonde commented 2 years ago

I now believe the problem is with GitHub and not with http.client or the proxy server.

{
  "args": {}, 
  "data": "{\"ref\":\"main\"}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Authorization": "Bearer someToken", 
    "Content-Length": "14", 
    "Host": "httpbin.org", 
    "X-Amzn-Trace-Id": "Root=1-6188dca6-253a33fc2677d14f015552e9"
  }, 
  "json": {
    "ref": "main"
  }, 
  "origin": "161.35.120.52", 
  "url": "https://httpbin.org/post"
}
{
  "authenticated": true, 
  "token": "someToken"
}
scotthansonde commented 2 years ago

I found my problem with the GitHub API. 😄 I installed my own instance of appserver and used it to proxy my request. I again got the 403 error, but I was able to see the error message returned in the data:

Request forbidden by administrative rules. Please make sure your request has a User-Agent header (http://developer.github.com/v3/#user-agent-required). Check https://developer.github.com for other possible causes.

(The request package apparently does not include a User-Agent header by default.)

I then added a User-Agent header to my request and the proxied request then succeeded! Drummer reported an error Can't read the URL, "https://api.github.com... because we received a status code of 204., but Status 204 is indeed a success (No Content).

scripting commented 2 years ago

@papascott -- incredible sleuthing! Thank you so much for tracking this down.

I see you've posted an issue on appServer, I think I know what it is. I'll respond over there.

scripting commented 2 years ago

@papascott -- do you think http.client should set the user-agent header if it's not already present?

I checked, in Frontier, tcp.httpClient does exactly this. A comment posted on 3/2/99:

image

scripting commented 2 years ago

Here's the OPML of the source of Frontier's tcp.httpClient for reference.

The change notes are in a comment at the top of the routine.

http://scripting.com/publicfolder/misc/tcp.httpclient.opml

scripting commented 2 years ago

Update: I made the change adding a User-Agent header which can be overridden by the caller, in http.client.

http://scripting.com/drummer/blog/2021/11/08/142219.html?title=moreWorkOnHttpclient

scotthansonde commented 2 years ago

I tried out the new deployed version in Drummer, my original script (not setting a User-Agent) worked , returning Status 204. No error message.

PostMonsterG commented 2 years ago

I'm unable to get this working in Drummer 2.0.19 in Safari on an iPad using the local client. It works OK when I set the Boolean to true to use the proxy server.

console.log( http.client( { url: "http://2005.opml.org/validator/test/encoding.txt" }, false ))

Here's the console output:

runScriptText: processedScriptText == (async function () {await processRunCursorScriptResult(await console.log(await http.client({ url: 'http://2005.opml.org/validator/test/encoding.txt' }, false)), 2);})() null

scripting commented 2 years ago

when you say you can't get it working, what happens?

PostMonsterG commented 2 years ago

I get the console output provided above. It just says "null". No error message, not the contents of the file as when using the proxy server.