Add Dropbox backend

[aug2uag]

Implement Dropbox as backend

references: @mrkurt Go implementation

[mrkurt]

Yes please!

[aug2uag]

i'll proceed with implementing /downloads API, it's a v1 API and no notes, i don't think it'll be any issue

[aug2uag]

@mrkurt would you please provide me some help with FlyRequest:

import { FlyRequest } from "@fly/v8env/lib/fly/fetch";
...
var fr = new FlyRequest()

ERROR in /usr/local/var/foss/cdn/src/backends/dropbox.ts(39,16)
      TS2693: 'FlyRequest' only refers to a type, but is being used as a value here.
    at /usr/local/var/foss/cdn/node_modules/@fly/build/lib/main.js:30:22

[mrkurt]

Ah, FlyRequest is really just an interface, which means it can't "created". You can safely use new Request(url) for everything.

[aug2uag]

@mrkurt I can't seem to stick the headers to request, would you please take a look, i think according to docs this should work no? the log output for the request is in the comments

/**
 * @module Backends
 */

import { ProxyFunction } from "../proxy";
// import { fetch } from "@fly/v8env/lib/fetch";

/** const Dropbox path */
const dropboxEndpoint = "https://content.dropboxapi.com/2/files/"

/**
 * Dropbox options
 */
export interface DropboxOptions {
    token: string,
    path: string
}

/**
 * Dropbox request options
 */
export interface DropboxRequestOptions {
    method: string,
    headers: Headers
}

/**
 * Creates a POST `fetch` with Dropbox headers
 * ```
 *  {
 *      "Authorization": `bearer ${options.token}`,
 *      "Dropbox-API-Arg": options.path
 *  }
 * ```
 * @param options strictly enforced DropboxOptions (no support for strings yet)
 */
export function dropbox(options: DropboxOptions): ProxyFunction<DropboxOptions> {

  const host = `${dropboxEndpoint}download` 

  const token = options.token
  console.log(token, '= token')

  async function proxyFetch() {

    var config = normalizeOptions(options)
    var request = new Request(host, config)

    console.log(request)
    console.log('\n')
    /** 
        ** empty headers: **

        { bodySource: null,
          stream: null,
          method: 'POST',
          url: 'https://content.dropboxapi.com/2/files/download',
          referrer: null,
          mode: null,
          credentials: 'omit',
          headers: { headerMap: {} } }  //<== empty headers here
     */

    let bresp = await fetch(request)

    console.log(bresp)

    return bresp
  }

  return Object.assign(proxyFetch, { proxyConfig: options })

}

function normalizeOptions(options: DropboxOptions): DropboxRequestOptions {
    var headers = new Headers()
    headers.append("Authorization", `bearer ${options.token}`)
    headers.append("Dropbox-API-Arg", options.path)

    return { 
        method: 'POST',
        headers: headers
    }
};

dropbox.normalizeOptions = normalizeOptions

[aug2uag]

sorry i'm so helpless here @mrkurt

[mrkurt]

Ah no worries! Sorry I missed your question.

I think what's happening here is that new Request(url, options) can't actually handle a Headers object type. Try changing normalizeOptions to this:

function normalizeOptions(options: DropboxOptions): DropboxRequestOptions {
    var headers = {
            "Authorization": `bearer ${options.token}`,
        "Dropbox-API-Arg": options.path
         }

    return { 
        method: 'POST',
        headers: headers
    }
};

[aug2uag]

@mrkurt for some reason the headers are stripped as RequestInit (how you did it with change in return type) and as object literal passed directly to Request (as viewed by bresp), i was thinking these may be culprits:

• headers are stripped, contradicted by setting headers on the GET requests of nearly all other backends
• something to do with the headerMap structure

The headers are just not being set:

    const headers = new Headers()
    const authValue = `bearer ${options.token}`
    const pathValue = options.path
    console.log(authValue) // bearer $TOKEN
    console.log(pathValue) // /foo/foo.foo
    headers.append("Authorization", authValue)
    headers.append("Dropbox-API-Arg", pathValue)
        console.log(headers) // { headerMap: {} }

[mrkurt]

Since Headers is a class, the console log doesn't know how to serialize it. Try console.log(headers.toJSON()). That's a utility method we added since opaque classes can be a pain sometimes.

[aug2uag]

@mrkurt you're right, JSON.stringify outputs the values and it shows the headers change from RequestInit to Request

function normalizeOptions(options: DropboxOptions): RequestInit {
    var headers = {
        "Authorization": `bearer ${options.token}`,
        "Dropbox-API-Arg": options.path
     }

    return { 
        method: 'POST',
        headers: headers,
        cache: 'default'
    }
};

var config = normalizeOptions(options)
console.log(JSON.stringify(config, null, 4))
/*
{
    "method": "POST",
    "headers": {
        "Authorization": "bearer x_Dfb5viqakAAAAAAAAGoUwH-_QTKeaKqGoN3iIM-M2capc3jsU0vx_HtZEQdZKu",
        "Dropbox-API-Arg": "/foo/foo.foo"
    },
    "cache": "default"
}
*/

var request = new Request(url, config)
console.log(JSON.stringify(request, null, 4))
/*
{
    "bodySource": null,
    "stream": null,
    "method": "POST",
    "url": "https://content.dropboxapi.com/2/files/download",
    "referrer": null,
    "mode": null,
    "credentials": "omit",
    "headers": {
        "authorization": [
            "bearer x_Dfb5viqakAAAAAAAAGoUwH-_QTKeaKqGoN3iIM-M2capc3jsU0vx_HtZEQdZKu"
        ],
        "dropbox-api-arg": [
            "/foo/foo.foo"
        ]
    }
}
*/

I'm getting these results in fly test by appending defs (subdomain_services_spec.ts:6) with

  { backend: dropbox, options: ["subdomain", "directory"], tests: [{
    token: 'x_Dfb5viqakAAAAAAAAGoUwH-_QTKeaKqGoN3iIM-M2capc3jsU0vx_HtZEQdZKu',
    path: "/foo/foo.foo"}
  ]}

My control is your original Dropbox backend in Go on a shared file I configured in Dropbox (I probably should delete the token by end of day today):

$ curl -X POST https://content.dropboxapi.com/2/files/download  \
--header "Authorization: Bearer x_Dfb5viqakAAAAAAAAGpOul3cDHvTUWczgaWlDWVWT-yTq2u6db2Y2hVuaDPvrm" \
--header "Dropbox-API-Arg: {\"path\": \"/foo/foo.foo\"}"

It's great news knowing headers are set, yet the test doesn't pass and I think it's because the headers are arrays in the Request object.

[aug2uag]

looking at the github backend. will give a crack at this over the weekend

[aug2uag]

@mrkurt please check out my findings and advise

I noticed in core/lib/bridge/fetch.js that protocol was being set to 'https:', I created a https environment and found out I need protocol not to contain the : in it.

If I set protocol for dropbox, i get a crash in getHeapStatisticsSync, and putting it in a try / catch causes a timeout and error message saying to call done().

/usr/local/var/foss/cdn/node_modules/@fly/core/lib/local_runtime.js:51
                log_1.default.info(`Runtime heap: ${(this.isolate.getHeapStatisticsSync().total_heap_size / (1024 * 1024)).toFixed(2)} MB`);

Here is the dropbox.ts backend under test:

/**
 * @module Backends
 */

import { ProxyFunction } from "../proxy";
// import { fetch } from "@fly/v8env/lib/fetch";

/** const Dropbox path */
const dropboxEndpoint = "https://content.dropboxapi.com/2/files/"

/**
 * Dropbox options
 */
export interface DropboxOptions {
    token: string,
    file: string,
    path?: string,
    hostname?: string
}

/**
 * Creates a POST `fetch` with Dropbox headers
 * ```
 *  {
 *      "Authorization": `bearer ${options.token}`,
 *      "Dropbox-API-Arg": options.path
 *  }
 * ```
 * @param options strictly enforced DropboxOptions (no support for strings yet)
 */
export function dropbox(options: DropboxOptions): ProxyFunction<DropboxOptions> {

  const host = `${dropboxEndpoint}download` 
    const token = options.token

  async function proxyFetch() {
        var config = normalizeOptions(options)
    let bresp = await fetch(host, config)
        return bresp
  }

  return Object.assign(proxyFetch, { proxyConfig: options })

}

function normalizeOptions(options: DropboxOptions): RequestInit {
    var headers = {
      "Authorization": `Bearer ${options.token}`,
        "Dropbox-API-Arg": options.file
   }

    return { 
        method: 'POST',
        headers: headers,
        cache: 'default',
    }
};

dropbox.normalizeOptions = normalizeOptions

-- UPDATE -- Sorry Kurt, false flag. It's my understanding of the proxy that's very off, my fault.

TypeError [ERR_INVALID_PROTOCOL]: Protocol "https" not supported. Expected "https:"

Still trying to figure this out!

[aug2uag]

Ok, so I'm very dumb. It was ( i ) not using capitalized 'B' in bearer and ( ii ) not putting the filename in { path : filename }

It's working now.