search

cupOJoseph / storj-web-gateway

A web gateway to the storj network.
https://storjgateway.com/
MIT License
4 stars 4 forks source link

Storj IPFS plugin #1

Closed cupOJoseph closed 5 years ago

cupOJoseph commented 5 years ago

Goal: create a plugin for IPFS on the server which runs the Storj configuration. Allow for a storj configuration profile during ipfs init.

Resources

Acceptance Criteria

Add a basic IPFS plugin to the project. The plugin can run with a storj configuration Please include some docs on how this works.

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 3000.0 STORJ (730.5 USD @ $0.24/STORJ) attached to it as part of the storj fund.

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

These users each claimed they can complete the work by 11 months, 1 week from now. Please review their action plans below:

1) zyfrank has been approved to start work.

I'm familiar with decentralized storage, I'd like to try this challenge

Learn more on the Gitcoin Issue Details page.

cupOJoseph commented 5 years ago

Thanks for applying @zyfrank. Is this issue clear with its goals? Find me in the gitcoin community slack for questions. Could you please message me there when you get a chance? Hoping to have this merged within the next 2 weeks if possible

zyfrank commented 5 years ago

@jschiarizzi By now, it is ok for me. Will let you know if anything is fuzzy during following implementation

cupOJoseph commented 5 years ago

@zyfrank how is it coming along? I need a progress update. I would like to have this section done by this weekend, do you think that will happen? let me know if you need help.

zyfrank commented 5 years ago

@jschiarizzi currently I'm trying extracting code(do real data put, get etc.) from storj uplink to ipfs storj datastore plugin, but I do not think this can be finished by this weekend

cupOJoseph commented 5 years ago

Okay, thanks for the update. No worries if it takes a bit longer, just doing our best here. Is there a specific part that you think could be done in parallel? We can change the scope and add another bounty for someone else to do that part

zyfrank commented 5 years ago

@jschiarizzi after investigation, my suggested solution is: on a clone of go-ipfs, I will clone go-ipfsunder my github account add support for go-ds-s3 that means change datastores.go(add logic for go-ds-s3) just as what https://github.com/ipfs/go-ipfs/blob/v0.4.18/repo/fsrepo/datastores.go does for go-ds-badger, go-ds-leveldb etc.

In this way, what you should do is: compile a new ipfs, compile go-ds-s3, then on a storj env. It should be ok for acessing storj by ipfs

How do you think about this solution?

cupOJoseph commented 5 years ago

Sounds like a great idea. I think that's similar to the rtrade version. If you end up having to get into the s3 parts a lot let me know and we can beef up the bounty a bit. Or if you think there is a part of this that could be done in parallel

We can merge it under a new branch here afterwards, or perhaps inside a folder here under /go-ds-s3. What ever is easier for you

zyfrank commented 5 years ago

My following working function well on my test env. (my work based on go-ipfs 0.4.18, go-datastore 3.4.0) 1 make little change to go-ds-s3(remove ec2 related code) 2 copy hacking from https://github.com/RTradeLtd/storj-ipfs-ds-plugin/tree/master/fsrepo (mainly change file gx\ipfs\QmPEpj17FDRpc7K1aArKZp3RsHtzRMKykeK9GVgn4WQGPR\go-ipfs-config\profile.go for storj plugin profile during ipfs init --profile storj and gx\ipfs\QmUJYo4etAQqFfSS2rarFAE97eNGB8ej64YkRT2SmsYD4r\go-ipfs\repo\fsrepo\datastores.go for plugin related hacking)

Here choice of solution will be for 1 create a new github repo (this can clone from go-ds-s3) and put the needed little change in this new repo you can create a null repo under your github account and tell me the link (suggested name is ipfs-ds-storj-s3c-plugin) or I can create this repo under my github account

for 2 clone go-ipfs to a new repo, based on tag v0.4.18, clone go-ipfs-config to a new repo, based on tag v0.2.16, may be we shoud publish a new gx package
changed mentioned two files
then you need not do copy during compile ipfs and it is easy to track the change ()

Hope to get your feedback

cupOJoseph commented 5 years ago

clone go-ipfs to a new repo, based on tag v0.4.18, clone go-ipfs-config to a new repo, based on tag v0.2.16, may be we shoud publish a new gx package changed mentioned two files then you need not do copy during compile ipfs and it is easy to track the change ()

This seems like the cleaner solution, although maybe a bit harder. If you think they will take about the same time then this is probably better. However, we are on a time crunch so getting minimal features working fastest is my priority.

zyfrank commented 5 years ago

if now unexpected problem，need another 2days or so

cupOJoseph commented 5 years ago

You should commit what you have so far

zyfrank commented 5 years ago

@jschiarizzi Hi, you can take a look at https://github.com/zyfrank/ipfs-s3c-storj-plugin

cupOJoseph commented 5 years ago

Okay, thanks. Will you need a storj "accessKey" and "secretKey" to test? Will we be able to finish this soon?

cupOJoseph commented 5 years ago

I will get some keys for you. You'll have to message me privately though. gitcoin community slack or email is fine. joseph@fourthwavestudios.com

Edit: I got your email, waiting on keys. Hopefully you can keep working on the core of this without it.

zyfrank commented 5 years ago

@jschiarizzi have sent an email to you. need testing key.

zyfrank commented 5 years ago

@jschiarizzi has any suggestion for this kind of solution?

cupOJoseph commented 5 years ago

Hey Frank, I just got an update from Storj system engineers.

They havent publicly added using APi keys for the main storj network, that launches later in April. For now we need to make a a version that connects with just a local s3 gateway:

https://github.com/storj/storj/wiki/Uplink-CLI

~/go/bin/uplink setup --api-key abc123 --satellite-addr 127.0.0.1:7778 \
  --enc-key highlydistributedridiculouslyresilient

api key is abc123 and the network has to be local 127.0.0.1:7778. Does this make sense?

cupOJoseph commented 5 years ago

I think that means we have to run an instance of the s3 gateway locally perhaps? https://github.com/storj/storj/wiki/S3-Gateway

zyfrank commented 5 years ago

@jschiarizzi acctually， I think https://github.com/storj/storj/wiki/S3-Gateway is not synchronized with https://github.com/storj/storj/wiki/Test-network

during my testing work， I use the method defined in https://github.com/storj/storj/wiki/Test-network command is storj-sim network run , this command will setup all satelite, gateway, storage node and corresponding identity for you. In this way, life is easy.

that means gateway listened on 127.0.0.1:9000, and you can find corresponding keys in ~/.local/share/storj/local-network/gateway/0/config when you use uplink you should use command just like this : "uplink --config-dir ~/.local/share/storj/local-network/gateway/0/ ls sj://ipfs"

zyfrank commented 5 years ago

@jschiarizzi any update on this?

cupOJoseph commented 5 years ago

Yes, sorry, I thought I replied yesterday, but something went wrong? thanks for being diligent

Let's run it locally on 127.0.0.1:9000 (or whatever port is needed) for now. And we will integrate with real api key and satellites later in April once that is publicly launched by storj.

zyfrank commented 5 years ago

Hi @jschiarizzi , I have made some refactor on the code, now https://github.com/zyfrank/ipfs-s3c-storj-plugin is built on go-ipfs 0.4.19 and go-datastore 3.6.1 A big good news is that now go-ipfs does not depend on our package. So you can ignore corresponding change contained in three repoes I cloned from ipfs.

BTW after I start storj local test env (built from storj master), I test ipfs add, ipfs cat, ipfs ls, ipfs get, and the tests are successful. I think this task is almost finished. what's your opinion?

cupOJoseph commented 5 years ago

Looks awesome! I have an event this evening so I can test it by tomorrow.

Can we add the commands needed to download everything and run it as is to the readme please?

zyfrank commented 5 years ago

You want a shell script？

cupOJoseph commented 5 years ago

That would be even better. I gave it its own issue #7,

https://gitcoin.co/funding/details/?url=https://github.com/jschiarizzi/storj-ipfs-gateway/issues/7

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work for 3000.0 STORJ (804.0 USD @ $0.27/STORJ) has been submitted by:

  1. @zyfrank

@jschiarizzi please take a look at the submitted work:

zyfrank commented 5 years ago

@jschiarizzi any feedback?

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

The funding of this issue was increased to 3250.0 STORJ (871.0 USD @ $0.27/STORJ) .

cupOJoseph commented 5 years ago

Hi Frank, sorry I got distracted with unexpected personal things this weekend.

The instructions to run it are a bit unclear:

（make sure backup ~/.ipfs/, then delete this dir）

clone this repo to your $GOPATH/src/github.com

cd $GOPATH/src/github.com/ipfs-s3c-storj-plugin

For someone not super familiar with Go this is challenging. make install will not work until this path is setup correctly. Could you take a minute to make the directions more clear please. Like exactly what commands would a user run. I added 250 Storj to the bounty for doing these docs.

is that fair, does that make sense? Thanks

zyfrank commented 5 years ago

Hi @jschiarizzi

I have recorded two tty sesstion to record setup and test process, you can take a look at it by replay these two records (refer to README update)

cupOJoseph commented 5 years ago

I'm still having a lot of problems running these scripts on a clean install of ubuntu. I think maybe its $GOPATH problems. Working on it now, once I can get it to run, we can finish this issue

cupOJoseph commented 5 years ago

I'm getting errors all over the place from folders not being where its expected. I'm trying to run the commands in the installers manually... I'll try checking out the tty now..

zyfrank commented 5 years ago

which kind of error?

cupOJoseph commented 5 years ago

Screenshot (197) Screenshot (196)

zyfrank commented 5 years ago

first check golang version should >= 1.11, use bash some suggestion: clone storj-ipfs-gateway into /home/gitcoiner/src/github.com dir instead of Desktop. I think your home dir is /home/gitcoiner under /home/gitcoiner make dir src/github.com before run this script, setup golang:

export GOPATH=/home/gitcoiner
export GOROOT=/home/gitcoiner/go
zyfrank commented 5 years ago

seems that your env GOPATH GOROOT has some problem

cupOJoseph commented 5 years ago

Okay let me try that now... Thank you for the patience, I'm not super familiar with this.

There is no go folder in that directory. When I make one it still messes up. Yes I have set GOPATH go env GOPATH gives me /home/gitcoiner and yet it's still confused.

zyfrank commented 5 years ago

You should set GOROOT also. if no "go" folder, create a new "go" folder under /home/gitcoiner then try it again

zyfrank commented 5 years ago

@jschiarizzi if your test machine have a public ip, i can ssh to it and make some check

zyfrank commented 5 years ago

@jschiarizzi what about the testing on the plugin?

cupOJoseph commented 5 years ago

Sorry, I had a big move this weekend. I'm back to finish this now, I can't get it running shortly I'll ask find some help. Let's get this bounty done and paid out

cupOJoseph commented 5 years ago

go install -v ./cmd/... just keep failing, which I realize is an issue with my ability to install storj, and not necessarily part of what we've added here. I'm running go 1.8.3, which is what install from apt install golang gives me. Perhaps I need a newer version?

zyfrank commented 5 years ago

@jschiarizzi you should upgrade go from 1.8.3 to 1.11(I use this version), because as I know storj require go module support (which is delivered from version 1.11 and above)

zyfrank commented 5 years ago

@jschiarizzi you can download binary from https://golang.org/dl/ directly, to install, pls refer to https://golang.org/doc/install

cupOJoseph commented 5 years ago

Okay, doing that now

zyfrank commented 5 years ago

Any problem, just let me know

cupOJoseph commented 5 years ago

Updated Go to 1.12.1, installed storj manually. I learned a lot by doing this, and can now expand the docs more to make easier for the next person. looks like we have this working.

Next step is merging https://github.com/zyfrank/ipfs-s3c-storj-plugin into this repo some how Leave that to me. paying out the bounty now

gitcoinbot commented 5 years ago
Go Developer ⚡️ A *Go Developer* Kudos has been sent to @zyfrank for this issue from @jschiarizzi. ⚡️ Nice work @zyfrank! Your Kudos has automatically been sent in the ETH address we have on file.