Although Data Is Persisted Across Node Reboots, It Does Not Persist In The Pin Store

[bonedaddy]

Been experimenting with this plugin running an IPFS node, and have noticed that although I may restart my IPFS node which is using this datastore, and the data does remain stored long-term, subsequent queries such as ipfs pin ls indicate that nothing is pinned, which doesn't match what I'm seeing within my S3 datastore minio.

I think this might be in part caused by https://github.com/ipfs/go-ds-s3/issues/2 but I'm not sure what the actual issue is.

[Stebalien]

So, that issue has actually since been fixed (I think). What's your datastore config look like?

We store the pinset in /local/pins. Is this object not present in your S3 bucket?

[bonedaddy]

here is the entire datastore config

    "Datastore": {
    "StorageMax": "10GB",
    "StorageGCWatermark": 90,
    "GCPeriod": "1h",
    "Spec": {
      "mounts": [
        {
          "child": {
            "accessKey": "C03T49S17RP0APEZDK6M",
            "secretKey": "q4I9t2MN/6bAgLkbF6uyS7jtQrXuNARcyrm2vvNA",
            "bucket": "go-ipfs-storj-4",
            "region": "us-east-1",
            "regionEndpoint": "http://127.0.0.1:9000",
            "rootDirectory": "",
            "type": "s3ds"
          },
          "mountpoint": "/blocks",
          "name": "s3ds",
          "type": "log"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "HashOnRead": false,
    "BloomFilterSize": 1000000000
  },

nope when i access the bucket I do not see anything resembling /local/pins

edit: strange, i'm attempting again and it's working and i can list the pinset across node restarts

edit-2: so to get familiar with the plugin system I basically tried writing my own version of this plugin, and kept everything but the configuration the same. Here's the NewS3Datastore function from this codebase

func NewS3Datastore(conf Config) (*S3Bucket, error) {
    if conf.Workers == 0 {
        conf.Workers = defaultWorkers
    }

    awsConfig := aws.NewConfig()
    sess, err := session.NewSession()
    if err != nil {
        return nil, fmt.Errorf("failed to create new session: %s", err)
    }

    creds := credentials.NewChainCredentials([]credentials.Provider{
        &credentials.StaticProvider{Value: credentials.Value{
            AccessKeyID:     conf.AccessKey,
            SecretAccessKey: conf.SecretKey,
            SessionToken:    conf.SessionToken,
        }},
        &credentials.EnvProvider{},
        &credentials.SharedCredentialsProvider{},
        &ec2rolecreds.EC2RoleProvider{Client: ec2metadata.New(sess)},
    })

    if conf.RegionEndpoint != "" {
        awsConfig.WithS3ForcePathStyle(true)
        awsConfig.WithEndpoint(conf.RegionEndpoint)
    }

    awsConfig.WithCredentials(creds)
    awsConfig.WithRegion(conf.Region)

    sess, err = session.NewSession(awsConfig)
    if err != nil {
        return nil, fmt.Errorf("failed to create new session with aws config: %s", err)
    }
    s3obj := s3.New(sess)

    return &S3Bucket{
        S3:     s3obj,
        Config: conf,
    }, nil
}

heres the modified one I wrote (i suspect this might be the source of the issue)

// NewDatastore is used to create our datastore against the minio gateway powered by storj
func NewDatastore(cfg Config) (*Datastore, error) {
    // Configure to use Minio Server
    s3Config := &aws.Config{
        // TODO: determine if we need session token
        Credentials:      credentials.NewStaticCredentials(cfg.AccessKey, cfg.SecretKey, ""),
        Endpoint:         aws.String(cfg.Endpoint),
        Region:           aws.String(cfg.Region),
        DisableSSL:       aws.Bool(cfg.Secure),
        S3ForcePathStyle: aws.Bool(true),
    }
    s3Session, err := session.NewSession(s3Config)
    if err != nil {
        return nil, err
    }
    d := &Datastore{
        Config: cfg,
        S3:     s3.New(s3Session),
    }
    return d, nil
}

[bonedaddy]

Well I feel a bit stupid, the modified version i was playing with to learn how it works was missing a crucial piece, when I declared my Batch function I missed including the workers so

// Batch is a batched datastore operations
func (d *Datastore) Batch() (ds.Batch, error) {
    return &dBatch{
        d:       d,
        ops:     make(map[string]dBatchOp),
    }, nil
}

needed to be:

// Batch is a batched datastore operations
func (d *Datastore) Batch() (ds.Batch, error) {
    return &dBatch{
        d:       d,
        ops:     make(map[string]dBatchOp),
        workers: d.Workers,
    }, nil
}

I'll close this since I seem to have solved my issue, and have now learned enough to make sense of the plugin and datastore system, woot!

[Stebalien]

Looking at that config, you're not storing your pinset on S3. Your current datastore tree looks like:

• Mount: this is a datastore that mounts sub-datastores in various namespaces
  • LevelDB - mounted at /
  • S3 - mounted at /blocks

That means blocks will be stored on S3 but everything else will be stored locally. Honestly, this is probably the best way to do it but it does mean that only blocks will be persisted to S3.

[woss]

this is still an issue. today i checked and the pinning doesn't work when mounted to the s3 bucket.

ipfs pin add CID fails with error `Error: failed to enqueue CID: shutting down

Can Protocol Labs officially support this plugin so we can rely on it? Thanks