search

RTradeLtd / Nexus

🦑 Nexus is the IPFS private network node orchestration and registry service for Temporal
https://dev.temporal.cloud/private
Apache License 2.0
14 stars 4 forks source link

TEM-155: network routing scheme #13

Closed bobheadxi closed 5 years ago

bobheadxi commented 5 years ago

:construction_worker: Purpose

See ticket. In a nutshell, we want to provide an interface as follows:

requests for networks go via HTTP to:

etc. etc. the orchestrator daemon should then route requests to the appropriate node.

this opens the door for a wide range of things, like improved availability and hosting across multiple servers, as well as access tracking and potentially limiting

:rocket: Changes

Testing

make testenv
make daemon

set up a node called test_network:

make new-network
make start-network

access the API:

export TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NDc3Nzg2MzcsImlkIjoidGVzdHVzZXIiLCJvcmlnX2lhdCI6MTU0NzY5MjIzN30.2oqQCym2mcyFl8mjOHoGNtK41SJLwX0xbWScDruDECQ"
# scheme: <addr>/network/<name>/<feature>/<request>
curl --header "Authorization: Bearer $TOKEN" 127.0.0.1:8080/network/test_network/api/v0/repo/stat

you should get a response:

{"RepoSize":55956,"StorageMax":100000000000,"NumObjects":15,"RepoPath":"/data/ipfs","Version":"fs-repo@7"}

TODO

:warning: Breaking Changes

hopefully none

bobheadxi commented 5 years ago

paging @postables for initial thoughts

bonedaddy commented 5 years ago

@bobheadxi sorry about the delay, didn't get a notification for some reason.

I think making ports private on 127.0.0.1 is an excellent idea, and should provide some good security measures as well. I think these are great changes to start implementing. I like being able to route to different networks will make management of these a lot easier, and avoid quirky port forwarding, reverse proxying that isn't built in,

I got an error trying out your example curl command after spinning up the network curl 127.0.0.1:8000/network/test_network/api/v0/repo/stat

2018-12-30T22:57:26.385-0800    INFO    daemon  daemon/daemon.go:103    spinning up server      {"version": "v0.3.1-21-gb6212ee", "host": "127.0.0.1", "port": "9111"}
2018-12-30T23:06:24.521-0800    INFO    delegator.requests      log/middleware.go:33    request completed       {"version": "v0.3.1-21-gb6212ee", "path": "/network/test_network/api/v0/repo/stat", "query": "", "method": "GET", "user-agent": "curl/7.58.0", "status": 404, "took": "36.388µs", "real-ip": "127.0.0.1:58470", "request-id": "dark/BAxIhXJo2k-000001"}

Not sure if its due to the network not being started as the start or the network creation didn't log anything on the orchestrator side

solidity@dark:~/go/src/github.com/RTradeLtd/ipfs-orchestrator$ ./ipfs-orchestrator --config config.example.json --dev dev ctl StartNetwork Network=test_network
[WARNING] dev mode enabled
bobheadxi commented 5 years ago

@postables did you explicitly start up a new network?

bonedaddy commented 5 years ago

@bobheadxi yea i did, ill try again tomorrow after pulling in your most recent updates.

bonedaddy commented 5 years ago

@bobheadxi running into an issue:

solidity@dark:~/go/src/github.com/RTradeLtd/ipfs-orchestrator$ ./ipfs-orchestrator -dev dev -config config.example.json db test_network
[WARNING] dev mode enabled

doesn't look like that gives any output, tried with ctl CreateNetwork and it gave a panic

solidity@dark:~/go/src/github.com/RTradeLtd/ipfs-orchestrator$ ./ipfs-orchestrator -dev -config config.example.json ctl CreateNetwork test_network
[WARNING] dev mode enabled
function CreateNetwork found
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x80 pc=0x509620]

goroutine 1 [running]:
github.com/RTradeLtd/ipfs-orchestrator/vendor/github.com/RTradeLtd/ctl.(*CTL).Exec(0xc000335b40, 0xdcd940, 0xc0000b2010, 0xc0000bc050, 0x2, 0x2, 0xdc75c0, 0xc0000b6008, 0x0, 0x0, ...)
        /home/solidity/go/src/github.com/RTradeLtd/ipfs-orchestrator/vendor/github.com/RTradeLtd/ctl/ctl.go:56 +0x360
main.runCTL(0x7ffd67bd3106, 0x13, 0xbd0001, 0xc0000bc050, 0x2, 0x2)
        /home/solidity/go/src/github.com/RTradeLtd/ipfs-orchestrator/ctl.go:43 +0x2ac
main.main()
        /home/solidity/go/src/github.com/RTradeLtd/ipfs-orchestrator/main.go:76 +0x260
bonedaddy commented 5 years ago

the test examples you gave in the PR work, this is EXCELLENT! :tada:

codecov-io commented 5 years ago

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@a156d5c). Click here to learn what that means. The diff coverage is 84.87%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master      #13   +/-   ##
=========================================
  Coverage          ?   84.69%           
=========================================
  Files             ?       27           
  Lines             ?     1195           
  Branches          ?        0           
=========================================
  Hits              ?     1012           
  Misses            ?      126           
  Partials          ?       57
Impacted Files Coverage Δ
ipfs/docker.go 100% <ø> (ø)
network/ports.go 100% <ø> (ø)
ipfs/node.go 100% <100%> (ø)
delegator/context.go 100% <100%> (ø)
config/config.go 100% <100%> (ø)
log/middleware.go 100% <100%> (ø)
delegator/proxy.go 100% <100%> (ø)
delegator/util.go 100% <100%> (ø)
registry/registry.go 100% <100%> (ø)
ipfs/client.go 69.6% <54.77%> (ø)
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update a156d5c...6fd99c6. Read the comment docs.

guardrails[bot] commented 5 years ago

No more findings on this branch. This means you fixed everything we detected earlier. Good job!! 👍🎉

We will keep this comment up-to-date as you go along and notify you of any security issues that we identify.

Happy with the results? Give your feedback.