Any update on this rpoject?

[roshanDD]

Dear friend

I see that your github is missing the functionality to retrieve access code from the Alexa Voice Service. It gave authentication fail every time. Any update on your code?

Thanks Randy

[roshanDD]

@GOLDELEC

[GOLDELEC]

@roshanDD Every Alexa request needs an access code(auth token). So writing some code to retrieve the access code first is necessary. In recently, I'm spending some time on the office Alexa sample app(written in java) project to figure out how to port them to the project in C++ here. If you are interesting, you can also try to do that. The office Alexa sample app project: https://github.com/alexa/alexa-avs-sample-app

[hchaudhary1]

@GOLDELEC Have you reviewed this tutorial? https://miguelmota.com/blog/alexa-voice-service-authentication/ https://miguelmota.com/blog/alexa-voice-service-with-curl/

[roshanDD]

@GOLDELEC @hchaudhary1 Thanks for your replies. It was the access token error. I got it solved. However, I still have one more question regarding the down link channel. Currently, I have been sucessfully submitting a GET command to the /v20160207/directives in order to establish the down channel. However, the connection to the AVS should be a long live connection (which means that we still need to ping it when it's idle, aka sending empty packet in order to maintain such connection).

I am wondering what is your thought on maintaining such connection. Does nghttp2 already provided such functionality or do we have to implement our own function in order to ping it when it's idle?

Here is the guide from the Amazon AVS AVS down channel

[GOLDELEC]

@hchaudhary1 I just read the AVS authentication tutorial you sended. Itl is very useful. Thank you, in this tutorial, we can know the full way to get an auth token.

[GOLDELEC]

@hchaudhary1 Nghttp2 does not provide such ping function, and we should implement it by ourselves.

From the AVS sample code, we can see that this part code is just sending empty ping frame every 5 minutes to keep connecting after establishing down channel.

    protected HttpConnectionOverHTTP2 newHttpConnection(HttpDestination destination, Session session) {
        scheduler.scheduleAtFixedRate(new ServerPing(session), INITIAL_PING_DELAY_IN_MINUTES,
                PING_INTERVAL_IN_MINUTES, TimeUnit.MINUTES);
        return super.newHttpConnection(destination, session);
    }

...
    /**
     * Task to send a PING frame over an open HTTP/2 Session.
     */
    private static class ServerPing implements Runnable {
        private Session session;

        private ServerPing(Session session) {
            this.session = session;
        }

        @Override
        public void run() {
            if (!session.isClosed()) {
                PingFrame frame = new PingFrame(false);
                session.ping(frame, Callback.NOOP);
            }
        }
    }

[roshanDD]

@GOLDELEC @hchaudhary1

Thanks for your reply!

I am currently trying to use nghttp2 to build a multipart message. The message should be strcuture as the following.

I should use the nghttp2_submit_request(here) function, with nghttp2_nv *nva as my HTTP/2 header, and nghttp2_data_provider *data_prd for payload. However, I still have two problem.

1. How do I format the Json file? (Should I use cJson.h?)

2. I know how to read my audio file into the buffer (second part), but how do I append the audio file at the back of Json, so I have a multi-part message?

[hchaudhary1]

1 - haven't used cJson myself... should work (any json library will work)

2 - have you tried saving the entire header+message+message into a file. Then open a socket, and send the file chunked?

[roshanDD]

@hchaudhary1

Thanks for your reply. In the curl implementation, they have something called curl_formadd, which will append the {message header + message content} to a multi-part message.

In the nghttp2 implementation, they dont have something similiar. All they have is the nghttp2_data_source_read_callback (here), which reads the data coming from a parameter of nghttp2_submit_request.

As you said, maybe I can do {header + message + message}.

But my question is, how do I create a message? (because each message also got its own message header as well).

[hchaudhary1]

Hmm.. I think you need to study the "multipart/form" See following: https://stackoverflow.com/questions/4526273/what-does-enctype-multipart-form-data-mean

There's also good examples here: https://developer.amazon.com/public/solutions/alexa/alexa-voice-service/docs/avs-http2-requests#examples

[roshanDD]

Thanks for your reply.

I understand what to put for the header and the payload. But I am having trouble to figure out which nghttp2 library function to use in order to create these two messages, and append them together.

From your link, it looks like you are suggesting me to follow their formating? I am still confused

[GOLDELEC]

@roshanDD High level nghttp2 wrappers can make it easier, for example: https://www.nghttp2.org/documentation/asio_http2_client.h.html

[roshanDD]

@GOLDELEC , still not sure how to append Json to the back of Audio file... by using the library function

[hchaudhary1]

@roshanDD , to append, you just write it... theres no magic api to sticht these together. you just write one after the other

[roshanDD]

Is it possible for me to describe what I was think in my source code? Please see below. Here, I am using nghttp2_data_provider to open up an audio file, and write to the buffer.

ssize_t data_prd_read_callback(
    nghttp2_session *session, int32_t stream_id, uint8_t *buf, size_t length,
    uint32_t *data_flags, nghttp2_data_source *source, void *user_data) 
{

  printf("[INFO] C ----------------------------> S (DATA post body), length:%zu\n", length);

  int fd = source->fd;
  ssize_t r;
// writting my opened audio file into buffer
  while ((r = read(fd, buf, length)) == -1 && errno == EINTR); 
  printf("stream_id:%d, nread:%zu\r\n", stream_id, r);
  return nread;
}

void submit_postAudio(http2_session_data *session_data) {
  int32_t stream_id;
  http2_stream_data *stream_data = session_data->stream_data[STREAM_ID_REQUESTAUDIO];
  const char *uri = stream_data->uri;
  const struct http_parser_url *u = stream_data->u;
  char byLength = 0;

  const nghttp2_nv hdrs[] = {
  MAKE_NV(":method", "POST"),   MAKE_NV_CS(":path", stream_data->path),
  MAKE_NV(":scheme", "https"), MAKE_NV("accept", "*/*"),    
  MAKE_NV_CS("authorization", stream_data->access_token),
  MAKE_NV("content-type", "multipart/form-data; boundary=abcdefg123")
  };

  fprintf(stderr, "Request headers:\n");
  print_headers(stderr, hdrs, ARRLEN(hdrs));

  int fileDescriptor = open ("/my_audio.wmv", O_APPEND);  // open my audio file 
  nghttp2_data_provider data_prd;
  data_prd.source.fd = fileDescriptor   // set the file descriptor 
  data_prd.source.ptr = NULL;
  data_prd.read_callback = data_prd_read_callback;

  stream_id = nghttp2_submit_request(session_data->session, NULL, hdrs,
                                     ARRLEN(hdrs), &data_prd, stream_data);
  if (stream_id < 0) {
    errx(1, "Could not submit HTTP request: %s", nghttp2_strerror(stream_id));
  }

  stream_data->stream_id = stream_id;
}

What confused me is 1) how do I add a header (a message header to be more specific) to the audio. 2) how do I append it after my Json file.

[roshanDD]

or should I create a message struct and copy the entire message onto the buffer?

struct message{
  string begin_boundary; 
  string header;
  string end_boundary; 
  char * file_pointer;

};

ssize_t data_prd_read_callback(
    nghttp2_session *session, int32_t stream_id, uint8_t *buf, size_t length,
    uint32_t *data_flags, nghttp2_data_source *source, void *user_data) 
{

  printf("[INFO] C ----------------------------> S (DATA post body), length:%zu\n", length);

  int fd = source->fd;
   message my_message;
   m1.begin_boundary = "abcdefg123"
   header = "Content-Disposition:form-data; name = \"metadata\""
   memcpy(buffer, &my_data, sizeof(data));

   memcpy(buffer, &my_data, sizeof(my_message));
  printf("stream_id:%d, nread:%zu\r\n", stream_id, r);
  return nread;
}

[roshanDD]

@hchaudhary1 @GOLDELEC Thanks so much for your reply. I am currently trying to send the audio (http2 multipart encoded message) to the Amazon Alexa. I have encoded my message in the format as specified by my previous post (the photo). However, I have trouble to actually stream the data over.

As you know the buffer of nghttp2_data_source_read_callback can only support up to 16KiB. My audio file alone is 38646 Byte, and I also have 320 Byte of encoded string (the json + boundary etc). Currently, I pass all of payload (3846 Byte of audio + 320 Byte of string) to the nghttp2_data_source_read_callback buffer (buf), it gave me the error of

***nghttp2_session_send failed: -902.

I am confused and want to ask you two questions.

1. For the nghttp2_data_source_read_callback, does it prepare one data frame at a time? How do I continuously write to the buf of the read_callback function so that it can continously resume and stream my payload over?

2. Can I simply memcpy all of the payload to the buf of the nghttp2_data_source_read_callback , and let the library function to split my data into multiple data frame. Does the library automatically do that?

3. How can I print the payload of the sent POST request. It it at the on_frame-send_callback?

Thanks

[GOLDELEC]

@roshanDD nghttp2_data_source_read_callback can be called several times until the EOF flag is set, so the size of post data is not limited to 16KB. The data copied to buf can't exceed the max desired length at every turn. You can do some test to check whether the you can get the payload in on_frame_send_callback.

About the header and post body, for example these of an Alexa recognize request, the header should contain

'Authorization': 'token',
'Content-Type': 'multipart/form-data; boundary= 'boundary string',
'Transfer-Encoding': 'chunked',

the body should contain

--boundary string\r\n
Content-Disposition: form-data; name="request"\r\n
Content-Type: application/json; charset=UTF-8\r\n\r\n
json data\r\n
--boundary string\r\n
Content-Disposition: form-data; name="audio"\r\n
Content-Type: audio/L16; rate=16000; channels=1\r\n\r\n
audio raw data
--boundary string--\r\n

The full doc of nghttp2_data_source_read_callback:

/**
 * @functypedef
 *
 * Callback function invoked when the library wants to read data from
 * the |source|.  The read data is sent in the stream |stream_id|.
 * The implementation of this function must read at most |length|
 * bytes of data from |source| (or possibly other places) and store
 * them in |buf| and return number of data stored in |buf|.  If EOF is
 * reached, set :enum:`NGHTTP2_DATA_FLAG_EOF` flag in |*data_flags|.
 *
 * Sometime it is desirable to avoid copying data into |buf| and let
 * application to send data directly.  To achieve this, set
 * :enum:`NGHTTP2_DATA_FLAG_NO_COPY` to |*data_flags| (and possibly
 * other flags, just like when we do copy), and return the number of
 * bytes to send without copying data into |buf|.  The library, seeing
 * :enum:`NGHTTP2_DATA_FLAG_NO_COPY`, will invoke
 * :type:`nghttp2_send_data_callback`.  The application must send
 * complete DATA frame in that callback.
 *
 * If this callback is set by `nghttp2_submit_request()`,
 * `nghttp2_submit_response()` or `nghttp2_submit_headers()` and
 * `nghttp2_submit_data()` with flag parameter
 * :enum:`NGHTTP2_FLAG_END_STREAM` set, and
 * :enum:`NGHTTP2_DATA_FLAG_EOF` flag is set to |*data_flags|, DATA
 * frame will have END_STREAM flag set.  Usually, this is expected
 * behaviour and all are fine.  One exception is send trailer fields.
 * You cannot send trailer fields after sending frame with END_STREAM
 * set.  To avoid this problem, one can set
 * :enum:`NGHTTP2_DATA_FLAG_NO_END_STREAM` along with
 * :enum:`NGHTTP2_DATA_FLAG_EOF` to signal the library not to set
 * END_STREAM in DATA frame.  Then application can use
 * `nghttp2_submit_trailer()` to send trailer fields.
 * `nghttp2_submit_trailer()` can be called inside this callback.
 *
 * If the application wants to postpone DATA frames (e.g.,
 * asynchronous I/O, or reading data blocks for long time), it is
 * achieved by returning :enum:`NGHTTP2_ERR_DEFERRED` without reading
 * any data in this invocation.  The library removes DATA frame from
 * the outgoing queue temporarily.  To move back deferred DATA frame
 * to outgoing queue, call `nghttp2_session_resume_data()`.
 *
 * By default, |length| is limited to 16KiB at maximum.  If peer
 * allows larger frames, application can enlarge transmission buffer
 * size.  See :type:`nghttp2_data_source_read_length_callback` for
 * more details.
 *
 * If the application just wants to return from
 * `nghttp2_session_send()` or `nghttp2_session_mem_send()` without
 * sending anything, return :enum:`NGHTTP2_ERR_PAUSE`.
 *
 * In case of error, there are 2 choices. Returning
 * :enum:`NGHTTP2_ERR_TEMPORAL_CALLBACK_FAILURE` will close the stream
 * by issuing RST_STREAM with :enum:`NGHTTP2_INTERNAL_ERROR`.  If a
 * different error code is desirable, use
 * `nghttp2_submit_rst_stream()` with a desired error code and then
 * return :enum:`NGHTTP2_ERR_TEMPORAL_CALLBACK_FAILURE`.  Returning
 * :enum:`NGHTTP2_ERR_CALLBACK_FAILURE` will signal the entire session
 * failure.
 */
typedef ssize_t (*nghttp2_data_source_read_callback)(
    nghttp2_session *session, int32_t stream_id, uint8_t *buf, size_t length,
    uint32_t *data_flags, nghttp2_data_source *source, void *user_data);

[Ann-eat-apple]

@GOLDELEC

I am currently trying to use your on_frame_recv_callback function to print out the message of the received header. For some reasons, I am only seeing [INFO] C ----------------------------> S (HEADERS) on the stdout. It looks like the code didn't get into inside the for-loop....

I am pretty sure there is header data (because I am receiving the the on_data_chunk_recv_callback, and I can print out the data of the payload). I am wondering if you have any idea how to fix this?

static int on_frame_recv_callback(nghttp2_session *session, const nghttp2_frame *frame, void *user_data) {
    size_t i;
    switch (frame->hd.type) {
    case NGHTTP2_HEADERS:
        if (frame->headers.cat == NGHTTP2_HCAT_RESPONSE) {
            const nghttp2_nv *nva = frame->headers.nva;
            struct Request *req;
            req = (struct Request *) nghttp2_session_get_stream_user_data(
                    session, frame->hd.stream_id);
            if (req) {
                printf("[INFO] C <---------------------------- S (HEADERS)\n");
                for (i = 0; i < frame->headers.nvlen; ++i) {
                    fwrite(nva[i].name, 1, nva[i].namelen, stdout);
                    printf(": ");
                    fwrite(nva[i].value, 1, nva[i].valuelen, stdout);
                    printf("\n");
                }
            }
        }
        break;
    case NGHTTP2_RST_STREAM:
        printf("[INFO] C <---------------------------- S (RST_STREAM)\n");
        break;
    case NGHTTP2_GOAWAY:
        printf("[INFO] C <---------------------------- S (GOAWAY)\n");
        break;
    default:
       printf("get frame type:%d", frame->hd.type)
        ;
    }
    return 0;
}

[roshanDD]

@GOLDELEC

I actually successfully packaged my audio + json into a single Http2 request. However, I am getting "sorry, I didn't get that " response from Alexa. I guess somewhere in my transmission (or packing the json + audio), errors have occurred.

Also, I didn't added 'Transfer-Encoding': 'chunked' . Do you think this might be issue?

[hchaudhary1]

@roshanDD are you using PCM 16kHz @ 16-bit signed, mono @ 10ms per chunck? also, would be great to share how you solved nghttp2 issues.

[roshanDD]

@hchaudhary1 , yes, I am using the format as described in your previous post. I am trying to send "What is your height?", and only "What is your" got sent. I used the method which described in my previous post. The data_pred provider callback function.

[yitingpiaoxu]

@roshanDD you are so great! very very great, now I just using esp32 communicate with AVS，but i can not send the multiple message, can you give me your code or sample multiple message code ,my mail is: yitingpiaoxu@sina.cn 。thank you very much!