🍑 ③ Set up API

[denistsoi]

given the PR for netlify open authoring - https://github.com/greatislander/selfdefined/pull/1/checks

I think the doubling down of using netlify serverless functions is more feasible ...

Edit: since looking at netlify functions - the starter tier is only 125K calls per month. That totals on average at 173 per hour - which is quite restrictive... for $25 pm - its 2M or 2777 per hour - https://www.netlify.com/pricing/#features

for glitch - we can get 4000 per hour on the free tier - i might start it with glitch first https://glitch.com/pricing

[aguywithcode]

Happy birthday @tatianamac I came here from your birthday tweet.

I have a keen interest in ontology, taxonomy, and natural language processing. This project intrigues me and I would love to contribute and help in any way I can.

I see there is a discussion about how best to design the API and storage for the dictionary as well as the schema. In the ontology space there is an XML based protocol for defining concepts and their relationships called RDF. There is also a W3C recommendation called JSON-LD that fills the same space. We are using it in the IoT space to make metadata around devices and physical locations. It could prove useful in defining dictionary terms and how they relate.

WRT storage, I would also recommend using a NoSQL storage platform not necessarily for flexibility but in order to provide more performance for retrieval.

I'd like to offer my experience with cloud architecture, serverless development, and nosql database design. I can help us get an initial environment setup on Azure for development and production. The free tier for azure functions includes 1 million executions per month. CosmosDb (the Microsoft NoSQL database that supports MongoDB API as well as a Graph API) has a similarly generous free tier. For reference, Troy hunt of HaveIBeenPwned.com discusses the costs of running his site on Cloudflare and Azure Functions "It's costing me 2.6c per day to support 141M monthly queries of 517M records." A lot of this is due to a great caching model and usage of serverless functions to save on the backend.

Azure functions supports JavaScript, Python, C#, and Java so it will accommodate whatever language the team is comfortable with. In addition the Azure API Management service will make it easier to provide managed APIs for partners who would consume the service at free as well as paid tiers.

What I'd advise is that we identify an MVP of the service (it looks like a prime candidate would be the Twitter bot @tatianamac was talking about). Design the API that would support the bot and build the first revision. Perhaps we can setup a regular cadence of weekly calls (come as you can) to get some traction on the solution.

[tatianamac]

Thank you, @aguywithcode ! What a helpful and thorough digest.

I've restarted the Slack channel, which I'd love for you to join!

To me, there are two potential MVPs:

1. The Twitter Bot: @SelfDefinedAppBot is the current Twitter handle I'm squatting on for this. That's definitely a possibility. I'm happy to write specs for that.
2. The Dictionary: Ideally, I'd like for the dictionary to also use the API.

I'm going to switch over and address your comments on the OpenAPI pull request now.

Anyone else who has been on this thread is welcome to join the Slack channel as well if you'd like to continue to work on this! ✨

[mjoynes-wombat-web]

Oscar, Amy, Kurt, Tatiana and myself have been discussing the API in the dev channel of Slack.

I mentioned using AWS. Amy mentioned using GCP.

However, Oscar and Tatiana would prefer to avoid Amazon, Google and Facebook due to issues with their ethics.

We continued discussing additional solutions that avoid these.

Oscar mentioned Algolia as a solution for searching. We've looked at using a managed database on DigitalOcean. Kurt has been using Netlify functions for search and this would fit as a potential solution for the API code.

Amy shared this article as a solution to keep the Serverless function within the DigitalOcean ecosystem.

[kkemple]

I'm hesitant to use Algolia as the free plan doesn't allow for much customization and I wonder what the results of search terms would be. I'd love to run a few different options locally (like elastic search) and compare against what Algolia provides.

[mjoynes-wombat-web]

@kkemple I can say that elastic search provides a lot of options for customization. I created an e-commerce site using it over the last year and they had a bunch of custom data access coming from their internal warehouse management software. I agree that it would be good to test different solutions out.

Maybe the first step would be to create a local database with the terms and testing it out with different platforms?

[ovlb]

I'm hesitant to use Algolia as the free plan doesn't allow for much customization and I wonder what the results of search terms would be.

We could apply for the OSS tier immediately?

I'd love to run a few different options locally (like elastic search) and compare against what Algolia provides.

Like that idea.

[mjoynes-wombat-web]

I'd love to run a few different options locally (like elastic search) and compare against what Algolia provides.

Do we want to have a standardized local environment for this? I think we won't really be using it much in development if we're going managed database, service and serverless function. Or do we want to divi up the different types of services and then present them?

[mjoynes-wombat-web]

For the future, if we're using Auth for clients, would Netlify's auth work for this? Would we use API keys?

[BrentonPoke]

Did anyone ever look into Linode to see if they would be able to provide anything for this project?

[mjoynes-wombat-web]

@BrentonPoke I don't believe so, at least among the people who talked about it yesterday. Linode wasn't brought up. Do you have some more context outside of what's in this issue? I'm new here so I'm not fully aware of all that's going on.

[tatianamac]

(As an aside, when we're reading to start building stuff for this, I'd ask that we open it up as a new repo within the selfdefined organisation so we keep the web app separate from the API. Eventually we will want the web app to be served by the API, but this way it keeps concerns separate.)

[mjoynes-wombat-web]

@tatianamac I was going to start on some local testing of this tomorrow night. If you create a repo I'm happy to push what I'm working on up so other people can review. Also going to try and live stream a bit, though it'll be my first time so we'll see how that goes.

[ovlb]

There you go: https://github.com/selfdefined/api

[mjoynes-wombat-web]

Should I create a new issue to continue this discussion over there?

[ovlb]

Let’s keep the discussion here for context, but the implementation in the new repo. I’ve created a pinned issue in the new repo sending folks who are interested to this thread.

[tatianamac]

@ovlb What do you think about transferring this issue over there?

(I don't have strong feelings or experience either way.)

[ovlb]

@tatianamac Ah, didn’t know this was possible. Sounds like a plan :)

[mjoynes-wombat-web]

Been thinking about this more and Craft CMS would handle the majority of the needs of the dictionary. It would only need to be extended to allow for submissions of new words and revisions of existing ones. With the approval process to do so. It handles all the user requirements I could think of, allows for localization/translation "sites". Provides an excellent interface for content management. It integrates through a plugin with Elastic search. It also has a headless graphql/rest api mode that allows for working with Eleventy.

The CMS is very extendable and I know we could create our own module to handle the submissions and revisions from community members.

I use it almost daily at my full time job and really like it. Thoughts?

[mjoynes-wombat-web]

So because Craft CMS handles a lot of the features we'd need I'm going to spin up a test environment in it. I can do that a lot quicker than a test MySQL instance. If this isn't something that will work I'm happy to stop but until I get some direction this is how I will proceed.

[ovlb]

@ssmith-wombatweb Sorry for not getting back to you for so long. I’m glad you started thinking about a CMS, since it is quite a logical conclusion to our needs. I’ve only limited experience with Craft (and years old for that matter), so I trust your judgement fully. In short: Go for it :)

[mjoynes-wombat-web]

@ovlb Great! I think I can setup the admin portion of the site fairly easily to show off. I have a couple additional explanations and questions around the backend that I've outlined/asked in these three issues if you get a chance to look at them.

https://github.com/selfdefined/api/issues/7 https://github.com/selfdefined/api/issues/8 https://github.com/selfdefined/api/issues/9