[question] error no such file or directory

[adam-hanna]

Not sure if a gh issue is the best place for this. If there is a better place to ask questions, please let me know.

I'm getting the following error in my console when I try to add blocks to storage: rename /home/adam/.c3/bloc/put-030046100 /home/adam/.c3/bloc/blocks/AEJRGQAVONZMVAZRM3WS74JSE2DDK3PDFNEHFBAPPVJM4ZM5DNKG7FF77AYU3PBSHCR5CXZSWWSDW45MIYN64HWE4YJ64RQYUDGK7ZY5NMR2Y.data: no such file or directory

The error in question is happening from the bservice.AddBlock(basicIPFSBlock) line.

I was hoping that flatfs.CreateOrOpen(...) would do everything it needed, but maybe I'm implementing it incorrectly?

If I $ mkdir /home/adam/.c3/bloc/blocks && touch /home/adam/.c3/bloc/blocks/AEJRGQAVONZMVAZRM3WS74JSE2DDK3PDFNEHFBAPPVJM4ZM5DNKG7FF77AYU3PBSHCR5CXZSWWSDW45MIYN64HWE4YJ64RQYUDGK7ZY5NMR2Y.data everything works.

Thanks for your help

The important snippets of my code in question: fsstore/fsstore.go

package fsstore

import (
    "os"
    "path/filepath"
    "runtime"
    "strings"

    flatfs "github.com/ipfs/go-ds-flatfs"
)

// New ...
func New() (*flatfs.Datastore, error) {
    path := "~/home/adam/.c3/"

    // expand tilde
    if strings.HasPrefix(path, "~/") {
        path = filepath.Join(userHomeDir(), path[2:])
    }

    var (
        shardFn *flatfs.ShardIdV1
        err     error
    )

    if err := createDirIfNotExist(path); err != nil {
        return nil, err
    }

    shardFn, err = flatfs.ReadShardFunc(path)
    if shardFn == nil || err != nil {
        shardFn = flatfs.Prefix(4)
        if err := flatfs.WriteShardFunc(path, shardFn); err != nil {
            return nil, err
        }
    }

    return flatfs.CreateOrOpen(path, shardFn, true)
}

func userHomeDir() string {
    if runtime.GOOS == "windows" {
        home := os.Getenv("HOMEDRIVE") + os.Getenv("HOMEPATH")
        if home == "" {
            home = os.Getenv("USERPROFILE")
        }
        return home
    } else if runtime.GOOS == "linux" {
        home := os.Getenv("XDG_CONFIG_HOME")
        if home != "" {
            return home
        }
    }
    return os.Getenv("HOME")
}

func createDirIfNotExist(path string) error {
    if _, err := os.Stat(path); os.IsNotExist(err) {
        return os.MkdirAll(path, 0757)
    }

    return nil
}

foo/foo.go

import (
    ...
    "myproject/fsstore"

    libp2p "github.com/libp2p/go-libp2p"
    bstore "github.com/ipfs/go-ipfs-blockstore"
    bfmt "github.com/ipfs/go-block-format"
    cid "github.com/ipfs/go-cid"
    bserv "github.com/ipfs/go-ipfs/blockservice"
    nonerouting "github.com/ipfs/go-ipfs-routing/none"
    "github.com/ipfs/go-ipfs/exchange/bitswap"
    "github.com/ipfs/go-ipfs/exchange/bitswap/network"
)
...
ctx := context.Background()

diskStore, err := fsstore.New()
if err != nil {
    return fmt.Errorf("err building disk store\n%v", err)
}

blocks := bstore.NewBlockstore(diskStore)

newNode, err := libp2p.New(ctx, libp2p.Defaults, libp2p.ListenAddrStrings(cfg.URI))
if err != nil {
    return fmt.Errorf("err building libp2p service\n%v", err)
}

nr, err := nonerouting.ConstructNilRouting(nil, nil, nil, nil)
if err != nil {
    return err
}

bsnet := network.NewFromIpfsHost(newNode, nr)
bswap := bitswap.New(ctx, bsnet, blocks)
bservice := bserv.New(blocks, bswap)

...

func Put(bytes []byte, c *cid.Cid) error {
    basicIPFSBlock, err := bfmt.NewBlockWithCid(bytes, c)
    if err != nil {
        return err
    }

    return bservice.AddBlock(basicIPFSBlock)
}

[Stebalien]

This may be related to https://github.com/ipfs/go-ds-flatfs/issues/22 and https://github.com/ipfs/go-ds-flatfs/issues/40. I highly recommend you try go-ds-badger.

[adam-hanna]

Great, thanks! Any reason you suggest badger vs leveldb?

[Stebalien]

IIRC, badger supports large objects better. However, I really don't know why we decided to go with switch to badger versus stick with leveldb.

[Stebalien]

Why this is happening: When we use this datastore, we make sure to "mount" it under /blocks using a special "mount" datastore. This means that all keys are rooted at /, not /blocks/ (as they are in this case). That flattens the directory structure.

We could fix this, but, IMO, it's not worth putting too much effort into this datastore as it will always be slow.

[adam-hanna]

Thanks for all of the help. I switched over to your level-db implementation and it was "plug and play"! Thanks so much!