Golle
commented
2 years ago Hi,
Thanks for showing interest in Gamecure! This is as you stated in the other Issue an early alpha or an MVP for Content Creators at GOALS and the way we work here (plastic, single repo for engine+game, jira, GCS). The plan from the start was to make it possible to extend Gamecure with other cloud providers, VCS etc. and it's something we definitely want to do.
Documentation is not there yet, but it's something that we keep improving.
Regarding S3 support I've got some news for you. This library is currently in development and we plan to replace the current go-longtail client with it. https://github.com/Golle/longtail
This lib will make it easy to create your own BlockStore implementations and support any cloud provider (or local storage) you have.
Here's a simplified example of how a Remote store could be implemented once we have have integrated this lib.
public class CustomRemoteBlockStore : IBlockStore
{
private static readonly HttpClient Client = new();
private readonly string _baseUri;
private readonly string _fileExtension;
public CDNRemoteBlockStore(string baseUri, string fileExtension)
{
_baseUri = baseUri;
_fileExtension = fileExtension;
}
public Task PreflightGet(ReadOnlyMemory<ulong> blockHashes)
{
// Preload any hashes if needed
return Task.CompletedTask;
}
public async Task<StoredBlock?> GetStoredBlock(ulong blockHash)
{
// Get the stored block from any storage (Azure, S3, GCS, network drive, local filesystem)
var blockname = $"0x{blockHash:x16}";
var result = await ReadBytesFromStore($"{_baseUri}/{blockname.Substring(2, 4)}/{blockname}{_fileExtension}");
return result != null ? new StoredBlock(result) : null;
}
public async Task<StoreIndex?> GetExistingContent(ulong[] chunkHashes, uint minBlockUsagePercent)
{
// Read the store index file
var result = await ReadBytesFromStore($"{_baseUri}/storage/store.lsi");
return result != null ? new StoreIndex(result) : null;
}
private async Task<byte[]?> ReadBytesFromStore(string uri)
{
var result = await Client.SendAsync(new HttpRequestMessage(HttpMethod.Get, uri));
if (result.IsSuccessStatusCode)
{
var bytes = await result.Content.ReadAsByteArrayAsync();
Logger.Trace($"Read from CDN url: {uri} statusCode: {result.StatusCode} elapsed: {timer.Elapsed.TotalMilliseconds} ms Total bytes: {bytes.Length}");
return bytes;
}
return null;
}
}It would be awesome if you could join our public discord https://discord.com/invite/goals and ping me and we can have a chat about your use case and how your team would benefit from using Gamecure.
mikeseese
Longtail supports both Google Cloud Storage and S3 buckets, so I believe adding S3 support could open doors to more than one S3 storage provider (not just AWS, assuming https://github.com/DanEngelbrecht/golongtail/issues/209 gets implemented). I personally use DigitalOcean as my cloud provider, so it'd be great to not be locked in to using Google also.
I've started working on an implementation in a fork