Improvements

[mifi]

Potential improvements discovered during #87 but not blocking its merge:

• [x] github action slack notification like we had in travis
• [x] Use github releases with release notes instead of changelog. (I normally use np for publishing npm modules as it reduces chances of forgetting something)
• [ ] disable travis builder
• [ ] Option to auto-download results files (downloadResults: true) for createAssembly? I already created an example for this, but it could be a nice feature to be integrated in the API. Then we could even create a simple transloadit CLI that allows people to process local files using predefined templates for different useful operations!
• [ ] allow specifying filename and contentType of uploaded streams? See form-data
  • [ ] Maybe merge the concept of filename and fieldname (it's a bit confusing)
• [ ] Show off more cool things in readme (visual examples? gifs)
  • [ ] More files under examples/
• [ ] Create types for all any fields in ts

Logs / Global Event Emitter

Could allow the user subscribe to:

• Log events
• onProgress event
• (Suggested by kvz:) I think it would be nice if there was an option: reporter. By default it would be false. It could be set to nodesdk.consoleLogger or a custom function (progressCb add default console logger). Is this still needed now that we have the debug module for logging?

If we decide to add an EventEmitter it could lead to breaking changes (e.g. removing onUploadProgress and onAssemblyProgress from method options and move them to event listener)

Tests improvements

• [x] run tests periodically (every 24h? integration tests against live api)
• [ ] some tests/code is holding resources causing Jest did not exit one second after the test run has completed after test finished. I have worked around with --forceExit however this is not recommended.
• [x] maybe replace localtunnel with another solution, as we will be causing load on their servers
  • https://github.com/anderspitman/awesome-tunneling#open-source-at-least-with-a-reasonably-permissive-license
  • https://news.ycombinator.com/item?id=30891494
  • https://github.com/rapiz1/rathole
  • https://hookdeck.com/pricing
• [ ] Move more tests from integration to unit to speed up and improve stability. Can use nock recording to auto create test responses
• [ ] If possible, convert tests to typescript and make jest automatically check types in the tests (instead of using tsd with a duplicated and incomplete type test file)

[mifi]

localtunnel is down, causing github actions to fail. I think we should find a better solution https://github.com/transloadit/node-sdk/runs/2021853285?check_suite_focus=true

[kvz]

As per our docs https://transloadit.com/docs/ we could use ngrok which has a business (model) behind it, but allows limited free use(?)

[mifi]

Turns out ngrok doesn't work for requests coming from services hosted on AWS, so we should update our docs at https://transloadit.com/docs/ also to instead use localtunnel or something else. Or maybe we can host our own similar tunnelling service (possibly with a whitelist from transloadit IP ranges).

https://github.com/inconshreveable/ngrok/issues/408

[kvz]

Or maybe we can host our own similar tunneling service (possibly with a whitelist from transloadit IP ranges)

I like this idea.. It could maybe even work agnostically/transparently for a few SDKs if we build in support for that :thinking:

(for now i am leaving a comment in our docs as well)

[mifi]

so localtunnel seems to be down most of the time and causing a lot of failed builds recently.

there's a myriad of different open source proxies we could set up and run, but we would have to set up and maintain a server.

there's also cloudflare that seems to support this as a service. which solution should we go for?

[kvz]

We could ask @calbrecht to set up an ngrok service on our us-east-1 observer on maybe? https://github.com/inconshreveable/ngrok/blob/master/docs/SELFHOSTING.md any security implications with that?

we could host it on ngrok.transloadit.net. or can it not be hosted on a subdomain?

[kvz]

You know what, if Cloudflare can solve the problem immediately, let's use that first, and consider selfhosting an upgrade after that for which we can then reserve more time?

[mifi]

I found something new called Cloudflare Quick Tunnels: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/run-tunnel/trycloudflare It means we can create these "demo" tunnels with the cloudflared cli, without any configuration or dns (or even authentication!) So I will try this first and see if it works ebef58b8ddbf338e6ece1e7b9f0a4850ae2ab4b1

[mifi]

Have now replaced localtunnel with cloudflare quick tunnels #143. The cloudflared CLI is a bit lacking quality: It doesn't report when the tunnel is actually ready, so I needed to

1. do a regex on its stderr to find the URL
2. poll a DNS query of the hostname until it's ready using node.js resolve4 api (not using the operating system's lookup, as it will poison the dns cache).
3. poll a HTTP request through the tunnel in order to know that the tunnel is "healthy" before we can use it in our tests.

After a lot of obscure issues I have finally managed to work this all out, so it works now! for those curious, the logic is in https://github.com/transloadit/node-sdk/blob/master/test/testserver.js and https://github.com/transloadit/node-sdk/blob/master/test/tunnel.js

Note that when cloudflared starts up it prints:

line 2022-08-01T17:34:37Z INF Thank you for trying Cloudflare Tunnel. Doing so, without a Cloudflare account, is a quick way to experiment and try it out. However, be aware that these account-less Tunnels have no uptime guarantee. If you intend to use Tunnels in production you should use a pre-created named tunnel by following: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps

So it's not for production, but for our tests it should work, unless they start blocking Github actions' IPs. If they do that, we can go ahead and setup the private tunnels instead, and that shouldn't be too much work because they also use the same cloudflared CLI.