Main.hx:38: Error#422: No content-length header found @ tink.http.Header.byName:91

[nanjizal]

Have struggled to get Haxe to https to Shopify https://github.com/HaxeFoundation/haxe/issues/7702

I managed to get GET working with Electron. https://gist.github.com/nanjizal/38c03f19a24bcd2b98f8f03c7e9ec0da

But POST was returning Html login page normally according to thier forum a result of setting cookies but as far as I was aware I was not sending any cookies. Anyway Kevin suggest I try tink_https but not getting that far ...

Main.hx:38: Error#422: No content-length header found @ tink.http.Header.byName:91

My current code is here. Am I using it correctly?
According to Shopify documentation the path for GET products and for setting a new product with POST are the same. The URL I am using is correct as it works with GET and I traced it out.

https://gist.github.com/nanjizal/6fe95ae96e77eeea652e39beb85e61ce

I can upload some of the typedef's created for processing shopify json I have created most of them - if that is useful, currently they have comments containing real data so would need cleaning prior to putting on github.

Shopify provide free trial if you wanted to test tink_http against popular https service.

https://www.shopify.co.uk/free-trial?

Help on this would be much appreciated. Would I be better testing against haxe c++ ? Current Haxe build installed ( Haxe Compiler 4.0.0-preview.5+83d9c11 on mac ).

I know haxe really well for graphics but quite fustrating when Python can do this stuff well and we can't, http is not an area I know well beyond some as3 use in the past so hoping someone with more expertise knows.

[kevinresol]

Can you post a curl command that works? I can help you translate to tink_http from there

[kevinresol]

Please ignore my previous comment.

I see you are using neko and its underlying Client doesn't support chunked encoding (yet), so it expects a content-length header in the response. Can you try to educate the shopify API to not use a chunked response?

[kevinresol]

Can you also try to print the response header so I can investigate? (note to remove the .all() call)

tink.http.Client.fetch( url, {
            method: POST,
            headers: [new HeaderField(CONTENT_TYPE, 'application/json')],
            body: data,
        })
          .handle(function(o) switch o {
            case Success(res):
              trace(res.header);
            case Failure(e):
              trace(e);
          });

[nanjizal]

kevin thanks for quick response. Basically if you run this in the browser it will give you error json. http://myshopify.com/admin/products.json

To actually communicate you have to provide key and shop. For instance Alex shopify staff posted this PHP curl stuff.

$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL, 'https://api_key:api_secret@shop.myshopify.com/admin/products.json');
$result = curl_exec($ch);
curl_close($ch);

$products_json = json_decode($result);

foreach($products_json->products as $product){
  /* Zero in on variants */
  foreach($product->variants as $variant){
    /* Do a thing, eg: */
    echo $variant->price;
  }
}

So effectively a real url GET or POST might look something like:

https://987uagoeuaoe0u09a8oeu9a8oeu908a:8987uaeuaoeuaeouaoeu8a7oe98u7a0@yourshopname.myshopify.com/admin/products.json

Specifics of the calls for say product are detailed here, and I am successfully looping through pages of products and GET ing them in the Electron test so paths used must be suitable.

https://help.shopify.com/en/api/reference/products/product

[nanjizal]

Sure will try that Kevin

[nanjizal]

Kevin I get the same error Main.hx:53: Error#422: No content-length header found @ tink.http.Header.byName:91

[nanjizal]

This is the suggested Python code

import shopify

# Replace the following with your shop URL
shop_url = "https://{API_KEY}:{PASSWORD}@{SHOP_NAME}.myshopify.com/admin"
shopify.ShopifyResource.set_site(shop_url)

# Create a new product
new_product = shopify.Product()
new_product.title = "Burton Custom Freestyle 151"
new_product.product_type = "Snowboard"
new_product.vendor = "Burton"
new_product.save()

# Update a product
new_product.title = "Burton Custom Freestyle 151 - Python Edition"
new_product.save()

[kevinresol]

Kevin I get the same error Main.hx:53: Error#422: No content-length header found @ tink.http.Header.byName:91

Possible to switch to nodejs (it uses a native http impelmentation under the hood) for a quick glance?

[nanjizal]

I can try do I need to use hxElectron or is there a light way?

[nanjizal]

actually I can try that in my other project setup...

[kevinresol]

you can just use plain nodejs

anyway, I just want to inspect the response header from their API. You can use plain PHP too.

[nanjizal]

statusCode : 401, 
reason : Unauthorized, 
protocol : HTTP/1.1, 
fields : [{
name : server, 
value : nginx
},{
name : date, 
value : Fri, 22 Feb 2019 13:43:37 GMT
},{
name : content-type, 
value : application/json; charset=utf-8
},{
name : transfer-encoding, 
value : chunked
},{
name : connection, 
value : close
},{
name : x-sorting-hat-podid, 
value : 78
},{
name : x-sorting-hat-podid-cached, 
value : 0
},{
name : x-sorting-hat-shopid, 
value : ********
},{
name : x-sorting-hat-privacylevel, 
value : default
},{
name : x-sorting-hat-featureset, 
value : default
},{
name : x-sorting-hat-section, 
value : pod
},{
name : x-sorting-hat-shopid-cached, 
value : 0
},{
name : referrer-policy, 
value : origin-when-cross-origin
},{
name : x-frame-options, 
value : DENY
},{
name : www-authenticate, 
value : Basic Realm="Shopify API Authentication"
},{
name : x-request-id, 
value : ***************************
},{
name : x-shopify-stage, 
value : production
},{
name : content-security-policy, 
value : frame-ancestors 'none'; report-uri /csp-report?
// specific details of my shop I passed ...

Unsure if there is more as off screen if you need I can try to set electron app larger.

[kevinresol]

Well I got what I needed:

name : transfer-encoding, 
value : chunked

The sad news here is that for targets that are using the SocketClient chunked encoding is not supported (yet). I think the implementation is not so hard but I just don't have the time to fix it right now...

But I think nodejs should work.

[nanjizal]

I did a split on one of the out put numbers to get some of the bottom of the data added *** where needed

x-sorting-hat-privacylevel: default
x-sorting-hat-featureset: default
x-sorting-hat-section: pod
x-sorting-hat-shopid-cached: 0
referrer-policy: origin-when-cross-origin
x-frame-options: DENY
www-authenticate: Basic Realm="Shopify API Authentication"
x-request-id: *****
x-shopify-stage: production
content-security-policy: frame-ancestors 'none'; report-uri /csp-report?source%5Baction%5D=create&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=****
x-content-type-options: nosniff
x-download-options: noopen
x-permitted-cross-domain-policies: none
x-xss-protection: 1; mode=block; report=/xss-report?source%5Baction%5D=create&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=*******
x-dc: ash,gcp-us-central1

[kevinresol]

Perhaps a quick workaround is to not depend on content-length at all (because we have connection: close in the request anyway). You may want to try replacing these lines with:

case Success(parsed): 
  cb(Success(new IncomingResponse(parsed.a, parsed.b)));

[nanjizal]

In nodejs and in neko I get pretty similar response, replaced sensitive strings with random digits.

Main.hx:51: HTTP/1.1 401 Unauthorized
server: nginx
date: Fri, 22 Feb 2019 13:57:45 GMT
content-type: application/json; charset=utf-8
transfer-encoding: chunked
connection: close
x-sorting-hat-podid: 78
x-sorting-hat-podid-cached: 0
x-sorting-hat-shopid: 11111111
x-sorting-hat-privacylevel: default
x-sorting-hat-featureset: default
x-sorting-hat-section: pod
x-sorting-hat-shopid-cached: 0
referrer-policy: origin-when-cross-origin
x-frame-options: DENY
www-authenticate: Basic Realm="Shopify API Authentication"
x-request-id: aeuao89oeua87oeu098a7oeu89a7eu980
x-shopify-stage: production
content-security-policy: frame-ancestors 'none'; report-uri /csp-report?source%5Baction%5D=create&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=aeuao89oeua87oeu098a7oeu89a7eu980
x-content-type-options: nosniff
x-download-options: noopen
x-permitted-cross-domain-policies: none
x-xss-protection: 1; mode=block; report=/xss-report?source%5Baction%5D=create&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=aeuao89oeua87oeu098a7oeu89a7eu980
x-dc: ash,gcp-us-central1

So I am not even sure it's getting that far.

[nanjizal]

as in a success is already failed

[kevinresol]

Looks like you got a 401. Did you miss the shopify credentials?

[nanjizal]

if I run revised SocketClient code with my original neko code Main.hx:41: Error#401: Unauthorized @ tink.http._Fetch.FetchResponseImpl.all:133

[nanjizal]

I just tried getting all products with GET in the Electron code using my old approach so the URL is still valid as it's same one I use with your POST code.

[kevinresol]

@back2dos do you know how credentials-in-url works? Is it part of the http spec or just a client-side sugar? Maybe we need to support it in either the OutgoingRequest class or in the Client implementations?

@nanjizal In the mean time you should try to properly use a Authorization header to specify your credentials (instead of passing them in the url)

new HeaderField(AUTHORIZATION, 'Basic ${Base64.encode(Bytes.ofString('$user:$pass'))}')

[nanjizal]

Kevin I have tried Tink on just doing a "GET" and it fails in both Neko and hxElectron.

        tink.http.Client.fetch( url ).all()
          .handle(function(o) switch o {
            case Success(res):
              trace(res.header.statusCode);
              var bytes = res.body.toBytes();
              // do whatever with the bytes
            case Failure(e):
              trace(e);
          });

This is the GET code that does work for me on hxElectron

package;// haxe.http;

import js.html.*;

#if js
enum abstract CustomRequest( String ) to String from String {
    var POST;
    var GET;
    var PUT;
    var DELETE;
}

class HttpJs extends haxe.http.HttpBase {
    //public static var log: String-> Void;
    public var async:Bool;
    public var withCredentials:Bool;
    var req:js.html.XMLHttpRequest;

    public function new(url:String) {
        async = true;
        withCredentials = false;
        super(url);
    }

    /**
        Cancels `this` Http request if `request` has been called and a response
        has not yet been received.
    **/
    public function cancel()
    {
        if (req == null) return;
        req.abort();
        req = null;
    }
    public override function request(?post:Bool) {
        post = post != null ? post : false;
        this.customRequest(post ? "POST" : "GET", post);
    }
    public  function request_( customRequest: CustomRequest ) {
        switch( customRequest ){
            case GET:
                this.customRequest( "GET", false );
            case _:
                this.customRequest( customRequest, true );
        }
    }
    public function customRequest( method: String, post: Bool ) {
        responseData = null;
        var r = req = js.Browser.createXMLHttpRequest();
        var onreadystatechange = function(_) {
            //log(' onreadystatechange ');
            if(r.readyState != 4)
                return;

            var s = try r.status catch(e:Dynamic) null;/*
            if (s != null && js.Browser.supported) {
                // If the request is local and we have data: assume a success (jQuery approach):
                var protocol = js.Browser.location.protocol.toLowerCase();
                var rlocalProtocol = ~/^(?:about|app|app-storage|.+-extension|file|res|widget):$/;
                var isLocal = rlocalProtocol.match(protocol);
                if (isLocal) {
                    s = r.responseText != null ? 200:404;
                }
            }
                */
            if(s == js.Lib.undefined)
                s = null;
            if(s != null)
                onStatus(s);
            if(s != null && s >= 200 && s < 400) {
                req = null;
                onData(responseData = r.responseText);
            }
            else if (s == null) {
                req = null;
                onError("Failed to connect or resolve host");
            }
            else switch(s) {
            case 12029:
                req = null;
                onError("Failed to connect to host");
            case 12007:
                req = null;
                onError("Unknown host");
            default:
                req = null;
                responseData = r.responseText;
                onError("Http Error #"+r.status);
            }
        };
        if(async)
            r.onreadystatechange = onreadystatechange;
        var uri = postData;
        if(uri != null)
            post = true;
        else for(p in params) {
            if(uri == null)
                uri = "";
            else
                uri += "&";
            uri += StringTools.urlEncode(p.name)+"="+StringTools.urlEncode(p.value);
        }
        try {
            if(post)
                r.open(method, url, async);
            else if(uri != null) {
                var question = url.split("?").length <= 1;
                r.open(method,url+(if(question) "?" else "&")+uri,async);
                uri = null;
            } else
                r.open(method,url,async);
        } catch(e:Dynamic) {
            req = null;
            onError(e.toString());
            return;
        }
        r.withCredentials = withCredentials;
        if(!Lambda.exists(headers, function(h) return h.name == "Content-Type") && post && postData == null)
            r.setRequestHeader("Content-Type","application/x-www-form-urlencoded");

        for(h in headers)
            r.setRequestHeader(h.name,h.value);
        r.send(uri);
        if(!async)
            onreadystatechange(null);
    }

    /**
        Makes a synchronous request to `url`.

        This creates a new Http instance and makes a GET request by calling its
        `request(false)` method.

        If `url` is null, the result is unspecified.
    **/
    public static function requestUrl(url:String):String {
        var h = new haxe.Http(url);
        // h.async = false;
        var r = null;
        h.onData = function(d){
            r = d;
        }
        h.onError = function(e){
            throw e;
        }
        h.request(false);
        return r;
    }
}

#end

Unfortunately I can't get this code to work with POST.

[nanjizal]

the jQuery approach is commented out because I was worried it might be the reason POST was failing but it does not seem to make any difference to GET or POST in relation to shopify.

[kevinresol]

Did you try setting the authorization header instead of putting the credentials in the URL?

[back2dos]

Credentials in URLs are a browser thing, that gets translated to a basic auth header. That said, I can see why supporting them is useful. I guess OutgoingRequest is the best place, because it's central and will thus work for all clients.

[nanjizal]

I added to my 'Shop' class a function for that which seems to create something ending in '=' so looks ok.

    public function getKeyPass(){
        return haxe.crypto.Base64.encode(haxe.io.Bytes.ofString('$apiKey:$password'));
    }

Then on Neko I ran this...

        var url = shop.constructPath( PRODUCT );
        url.print();
        var pass = shop.getKeyPass();
        pass.print();
        var data = dummyProduct();
        data.print();
        tink.http.Client.fetch( url, {
            method: POST,
            headers: [new HeaderField(CONTENT_TYPE, 'application/json'),new HeaderField(AUTHORIZATION, 'Basic $pass')],
            body: data,
        }).all()
          .handle(function(o) switch o {
            case Success(res):
              trace(res.header.statusCode);
              var bytes = res.body.toBytes();
              // do whatever with the bytes
            case Failure(e):
              trace(e);
          });

The returned error changed from 401 to 400.

Main.hx:41: Error#400: Bad Request @ tink.http._Fetch.FetchResponseImpl.all:133

I then tried modifying constructPath

    public
    function constructPath( resource_: String, ?extra_: String ): String {
        var resource = resource_;//$apiKey:$password@
        var out: String = 'https://$name.myshopify.com/admin/$resource.json';
        if( extra_ != null ) out = out + extra_;
        log( out );
        return out;
    }

but this also give error 400.

[nanjizal]

@dmcblue created the working GET code, perhaps he has some input? https://github.com/HaxeFoundation/haxe/issues/7762

[nanjizal]

I have uploaded the current typedef's I am using incase anyone is feeling adventurous still needing more work. Not looked further at POST problem. https://github.com/nanjizal/hxTShopify

[kevinresol]

You should inspect the header and body of the 400 response and see if there are any info about the error

[dmcblue]

So there are a lot of changes from the top of this thread to here. If I understand this correctly, you are trying to make the 'Create a new Product' request.

1. The POST is failing, but GET requests are working, correct?
2. Also, do you have 'Authorization' header working in the GET requests or just the 'in-url-credentials'?

[nanjizal]

Unfortunately was not able to do more work on it this weekend but just to clarify.

in https://github.com/nanjizal/hxTShopify

1. the GET works but not the POST.
2. Just in url credentials

To test you need to create your Shop so like this but with real data. https://github.com/nanjizal/hxTShopify/blob/master/src/hxTShopify/store/DemoStore.hx

Then install and setup hxElectron and modify the App class https://github.com/tong/hxelectron/blob/master/demo/src/App.hx

So for instance to get all products and look for specific ones say with a 'cool' or 'COOL' tag - and print out the inventory levels of variants you might do something like:

    static var shop: Shop;
    static var explore: Explore;
    public static var logDiv: DivElement;
    static var div: DivElement;
    static function main() {
        window.onload = function() {
            div = cast document.getElementById( 'info' );
            logDiv = document.createDivElement();
            div.appendChild( logDiv );
            screenLog( 'test' );
            test();
        }
    }
public static function test(){
   var shop = new Shop( new LikeDemoStore() );
   var explore = new Explore( shop );
   explore.screenLog = screenLog;
   explore.productsLoadProgress = function( percent: Float )  screenLog( Std.string( percent ) + '%' );
   explore.productsLoaded = function( products: Array<Product> ){
            var str: String = '';
            var product: Product;
            var variant: Variant;
            for( i in 0...products.length ){
                product = products[ i ]; 
                if( product.tags.indexOf( 'cool' ) != -1  || product.tags.indexOf( 'COOL' ) != -1 ){
                str += product.title + ': ';
                    for( j in 0...product.variants.length ){
                        variant = product.variants[ j ];
                        str += variant.title + '( ' + variant.inventory_quantity + ' ), ';
                    }
                    str += '\n';
                }
            }
            str = str.substr( 0, str.length - 2 );
            screenLog( str );
      }
      explore.getProducts();
      // explore.createProduct();  // Test with POST fails.
}
public static function screenLog( d: Dynamic ){
        logDiv.innerText += ' ' + d + '\r\n';
} 

Ideally I should create a string abstract for tags with a 'to' Array method.

[dmcblue]

Perhaps its easier to simplify out all potentially confusing factors. The below is basically the simplest POST request code I can think of for this request. If it works, the issue is elsewhere your repository or another package like hxElectron. Otherwise, the output headers and response will help us understand what's wrong with the request itself. Either way, we'll be able to pinpoint the problem better.

package test;

import haxe.Http;
import haxe.Json;
import haxe.io.BytesOutput;

class Test {
    static function main() {
        var username = "";
        var password = "";
        var shopname = "";
        var url = 'https://$username:$password@$shopname.myshopify.com/admin/products.json';

        var payload = {
            "product" :{
                "title": "Burton Custom Freestyle 151",
                "body_html": "<strong>Good snowboard!</strong>",
                "vendor": "Burton",
                "product_type": "Snowboard",
                "published": false
            }
        };

        Test.post(
            url,
            payload
        );
    }

    static public function post(url:String, payload:Dynamic) {
        var req:Http = new Http(url);

        req.setPostData(Json.stringify(payload)); 
        req.addHeader("Content-type", "application/json");

        req.onError = function(error:String) {
            throw error;
        };

        req.onStatus = function(status:Int) {
            trace(status);
        };

        req.onData = function(data:String) {
            trace(data);
            trace(req.responseHeaders);
        };

        req.request(true);
    }
}

[nanjizal]

I tried your code in my Explore.hx adding this ( the url is same as GET so should be fine? ).

    public function createProduct(){
        var url = shop.constructPath( PRODUCT );
        var payload = {
            "product" :{
                "title": "Burton Custom Freestyle 151",
                "body_html": "<strong>Good snowboard!</strong>",
                "vendor": "Burton",
                "product_type": "Snowboard",
                "published": false
            }
        };
        post( url, payload );
    }
    public function post( url: String, payload: Dynamic) {
        var req: Http = new Http(url);
        req.setPostData(Json.stringify(payload)); 
        req.addHeader("Content-type", "application/json");      
        req.onError = function(error:String) {
            throw error;
        };
        req.onStatus = function(status:Int) {
            log( Std.string( status ) );
        };
        req.onData = function(data:String) {
            log( data );
            // log(req.responseHeaders);  // no responseHeaders for js target!!
        };
        req.request( true );
    }

I am getting 401, if I do it how I was it return 200 but only the html login page as documented above.

I thought I had to use custom for 'https'? What target is the standalone ment for, if nodejs how do I run that?

[kevinresol]

401 means you are authenticated, did you try constructing the authorization header manually? Also, please stop using credentials-in-url from now on, so that we can rule out that factor

[nanjizal]

Kevin I read this https://help.shopify.com/en/api/getting-started/authentication/private-authentication and your comment makes more sense to me, seems if I leave the pass on the url and add to head as well it now works!!

    public function createProduct(){
        var payload = {
            "product" :{
                "title": "Burton Custom Freestyle 151",
                "body_html": "<strong>Good snowboard!</strong>",
                "vendor": "Burton",
                "product_type": "Snowboard",
                "published": false
            }
        };
        post( shop.constructPath( PRODUCT ), payload, shop.getKeyPass() );
    }
    public function post( url: String, payload: Dynamic, pass: String ) {
        var req: Http = new Http(url);
        req.setPostData(Json.stringify(payload)); 
        req.addHeader("Content-type", "application/json");
        req.setHeader('Authorization', 'Basic $pass');
        req.onError = function(error:String) {
            throw error;
        };
        req.onStatus = function(status:Int) {
            log( Std.string( status ) );
        };
        req.onData = function(data:String) {
            var productHolder: ProductHolder = haxe.Json.parse( data );
            var product: Product = productHolder.product;
            log( Std.string( product ) );
        };
        req.request( true );
    }

Thank everyone this has been driving my crazy, and I was not able to dig in to it this weekend.

I will experiment with tink_http and see if I can use it insead, as I need PUT and DELETE. @dmcblue can you advise on PUT and DELETE?

Will update repo tomorrow ( today ) well in the morning. Must look at GET it can probably be simplified in my code.

[nanjizal]

I have DELETE, PUT working now, https.hx, perhaps I should rename. Do the methods look compatible with tink_http approaches? I have added use case in comments. To get ManyPages.hx - for product, customer or order pages, I have simplified explore and rebuilt Section. At moment code is only suited to nodejs unfortunately.

[kevinresol]

I wrote a PoC shopify API with tink_web here, the library itself is only about 100 LoC to enable calling the "List Products", "Create a Product" and "Delete a product" API.

It is all about writing a Haxe interface file to describe the API. And tink_web will do all the heavy-lifting. https://github.com/kevinresol/shopify/blob/28ef41f/src/shopify/Products.hx

Here is how the Haxe code usage looks like: (It is all fully typed. I included a $type directive to show that)

Create Product:

shopify.admin().products().create({
    product: {
        title: 'Burton Custom Freestyle 151',
        body_html: '<strong>Good snowboard!</strong>',
        vendor: 'Burton',
        product_type: 'Snowboard',
        tags: 'Barnes & Noble, John\'s Fav, "Big Air"',
    }
})
    .handle(o -> switch o {
        case Success(res):
            trace($type(res.product));
        case Failure(e):
            trace(e);

Delete product:

shopify.admin().products().delete(1746220482594)
    .handle(o -> switch o {
        case Success(res):
            trace(res);
        case Failure(e):
            trace(e);
    });

List Products:

shopify.admin().products().list({})
    .handle(o -> switch o {
        case Success(res):
            trace(res);
            for(product in res.products) trace($type(product));
        case Failure(e):
            trace(e);
    });

[kevinresol]

Let's call this done. The original issue will be tracked by #27