Discussion about merging/porting Diplomat into this library

[kiennt]

Hi @sashaafm

As I told in email, currently I am forking Diplomat library at

https://github.com/kiennt/diplomat

This library is already support Google Datastore

Could you please check out this library and give me some comments/thoughts?

[sashaafm]

Hi @kiennt thank you for answering. I'm divided. In a way it would be great since it seems to be well implemented and would make it a lot easier to implement in GCloudex. On the other hand it would break the actual pattern used in this project and would create the need for more external dependencies.

Do you have any experience using Diplomat?

I'll have to give this some more thought in the coming weeks since I'll be very busy until the start of July but we'll keep talking.

[kiennt]

@sashaafm

I am using diplomat in my project

Here is some example of how to use diplomat

# get objects from query
defmodule Tkn.AppSettingsController do
  use Tkn.Web, :controller

  def index(conn, _params) do
    settings =
      "SELECT * FROM AppSettings"
      |> Query.new
      |> Query.execute
      |> List.first
      |> Entity.properties
    conn |> render(:index, settings: settings)
  end
end

# complex example

  def fetch_invitation(item, users) do
    item
    |> Map.put("user", users[item["user_key"]])
    |> Map.put("inviter", users[item["inviter_key"]] |> Map.take(["id", "name", "image_url"]))
    |> Map.take([
      "id",
      "user",
      "inviter",
      "status",
      "shared_token_key",
      "expire_date_time",
      "circle_start_date_time",
      "circle_end_date_time",
      "circle_status",
      "created_at",
      "updated_at",
    ])
  end

  def show_invitations(conn, %{"circle_id" => circle_id}) do
    {id, _} = Integer.parse(circle_id)
    invitations =
      "SELECT * FROM Invitation WHERE circle_key = @1"
      |> Query.new([Key.new("Circle", id)])
      |> Query.execute
      |> Enum.map(&Entity.properties/1)

    users =
      invitations
      |> Enum.reduce([], fn item, acc ->
        acc ++ [item["user_key"], item["inviter_key"]]
      end)
      |> Key.get
      |> Enum.map(fn item -> {item.key, Entity.properties(item)} end)
      |> Enum.into(%{})

    result =
      invitations
      |> Enum.map(&fetch_invitation(&1, users))
    conn |> render(:show_invitations, invitations: result)
  end

I think if you want to implement API client for Datastore, the current pattern of this library is not suitable. Because API for datastore is RPC. It also supports REST API, but IMHO RPC is more efficient.

[1337hax]

But currently diplomat uses v2 API, which is around 2 times slower than v3. And as far as I read in diplomat repo, it's blocked because of lack of support of some protobuff features in their dependencies. So I think this is quite huge argument against diplomat in its' current version.

[sashaafm]

@1337hax yes I prefer that the API is implemented natively in GCloudex. That way it's less one dependency and we get a tighter control on the implementation.

[peburrows]

FYI, I released v0.1.0 of diplomat last night which switches to the v1beta3 API (@kiennt was a big help in eliminating blockers in the exprotobuf dependency).

[kiennt]

Hi guys

I think Diplomat 's implementation is good at this point.

But after some time using GAE on my app, I think we should move to new implement which works like a Ecto Adapter for GAE. And Diplomat could be a library to connect to GAE.

What do you guys think?

On Wed, Aug 10, 2016 at 6:37 AM, Phil Burrows notifications@github.com wrote:

FYI, I released v0.1.0 of Diplomat last night which switches to the v1beta3 API (@kiennt https://github.com/kiennt was a big help in eliminating blockers in the exprotobuf dependency).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sashaafm/gcloudex/issues/3#issuecomment-238725089, or mute the thread https://github.com/notifications/unsubscribe-auth/AAXSC2n9PoEpU7JvrI3_rHXJ4kjq21kEks5qeQ85gaJpZM4I5Ig5 .

Best regards Kien Nguyen Trung

[peburrows]

Ecto support is actually the next big thing on the roadmap for Diplomat. There's a decent amount of work to do there, but I'm currently working on building an Ecto adapter within Diplomat.

[sashaafm]

An Ecto adapter for Datastore and Bigtable is definitely interesting! @peburrows is this going to be done inside Diplomat or will it be an external project?

[peburrows]

My current plan is to implement it within Diplomat. Once I get into it a bit more, I might change my mind about that, but that's the direction I'm headed.

[kiennt]

@perburrows could you share with us more detail about your plan of support Ecto Adapter.

This is a really interesting.

On Fri, Aug 12, 2016 at 2:34 AM, Phil Burrows notifications@github.com wrote:

My current plan is to implement it within Diplomat. Once I get into it a bit more, I might change my mind about that, but that's the direction I'm headed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sashaafm/gcloudex/issues/3#issuecomment-239267297, or mute the thread https://github.com/notifications/unsubscribe-auth/AAXSC7D-Y_qoBCZsl58eLZldPMMHEBQ6ks5qe3lTgaJpZM4I5Ig5 .

Best regards Kien Nguyen Trung

[peburrows]

@kiennt my ultimate goal is to provide full Ecto support for all features of Ecto that Datastore supports. I've only just begun the process, so there's plenty of work to do, but eventually, support should be thorough enough that you can use Diplomat like any other Ecto adapter.

At a high level, this means a few things:

• You will be able to build and issue Ecto queries just like you would using any other adapter
• Diplomat will map Ecto changesets and schemas to their underlying Diplomat.Entity representations
• Transaction & Ecto.Multi support, of course
• Migrations will most likely be limited to creating indexes (since there is no real concept in Datastore of tables or such that need to be created)

[1337hax]

OK, now google data store has v1 stable REST api, i think supporting it would be very good. Any library has any plans for it ?

[sashaafm]

@1337hax I agree, it should be supported. Diplomat will support it, I guess. I have GCloudex in the backburner right now, I can't really devote time to it this month.

[peburrows]

@1337hax, yeah, Diplomat will soon support the v1 API (it's currently on the most recent beta, v1beta3, so switching to the official v1 should be no problem).

Somewhat related, I've also started work on an HTTP/2 client (it's called River) which is the first step toward the eventual goal of switching Diplomat (and my other GCP libraries) to use the gRPC APIs.

[kiennt]

@peburrows nice to know you are starting a project with HTTP/2 client.

FYI, I also started a project related to HTTP/2 (but currently only HPACK part is imlemented). https://github.com/kiennt/http2

Looking forward to see news from River and Diplomat.