RFC: Serverless APIs and functions

[dnys1]

This RFC is intended to collect feedback on Celest’s developer experience around serverless APIs and functions. We are sharing it as early as we can to give you the ability to provide feedback about what we’re building. Please take this opportunity to share your thoughts. Your input will help tremendously in making sure we deliver an experience you'll be delighted to use.

We will cover 3 main topics:

1. The Celest CLI used for local iterations/deployment
2. Defining serverless APIs and cloud functions
3. Setting up middleware and environment variables.

Getting started with Celest

Prerequisites

To use Celest in your Flutter app, you need the following prerequisites:

1. Install Flutter
2. Start a new fluter project using the flutter create command
3. Install the Celest CLI using a shell command

[!NOTE] This is an example and will not work just yet. Join the waitlist to get access when we launch!

$ curl --proto '=https' --tlsv1.2 https://install.celest.dev | sh

That’s it! You do not need any additional tooling to build, test, and deploy your backend.

Setting up the Celest CLI

After installing the Celest CLI, navigate to the root of your Flutter project and run the following command.

$ celest start

You will be prompted to sign in using GitHub. Once the authentication with GitHub is successful, a watch command will continue to run in your CLI to detect changes made to your Celest backend definition and code-generate a Dart client for you in the following path <flutter_app>/lib/celest/client.dart to test your changes locally. We will cover later how to use the code-generated client after defining your APIs and cloud functions.

The CLI will also create a folder in your project called celest, which will include the following files.

flutter_app/
└── celest/
    ├── apis/
    │   ├── greeting.dart # a serverless API definition
    │   └── middleware.dart # middleware definitions
    └── config/
        └── env.dart # environment variables

Creating serverless APIs and cloud functions

Creating serverless APIs and functions with Celest enables you to connect and aggregate information from different parts of your backend, and build custom business logic that runs completely in the cloud. You define your cloud functions as Dart functions, and Celest takes care of setting up and managing the backend infrastructure around them.

To get started with building your first API, navigate to the <flutter_app>/celest/apis/ folder and create a file named <api_name>.dart. You can create as many API files as you want in this directory.

[!TIP] Access to your APIs is denied by default. What this means is that you’ll need to add the @api.anonymous() annotation to the top of the file for APIs to be publicly accessible.

// Enables public access to the API.
@api.anonymous()
library;

import 'package:celest/api.dart' as api;

Future<String> sayHello(
  FunctionContext context,
  String name,
) async {
  return 'Hello, $name';
}

Future<String> sayGoodbye(
  FunctionContext context,
  String name,
) async {
  return 'Goodbye, $name';
}

The above code snippet is all you need to define your cloud functions! When the celest start command runs, a local environment is spun up and a Dart client is generated to help you connect to the local backend.

Below is an example of how you would use the generated client in your main.dart file.

import 'package:flutter/material.dart';
import 'package:flutter_app/celest/client.dart' as celest;

void main() {
  celest.init();
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return FutureBuilder(
      // Call your cloud function like a normal Dart function!
      future: celest.apis.greeting.sayHello('Celest'),
      builder: (_, snapshot) => switch (snapshot) {
        AsyncSnapshot(:final data?) => Text(data),
        _ => const CircularProgressIndicator(),
      },
    );
  }
}

Using middleware for your APIs and cloud functions

Middleware enables you to have logic that can run before and/or after your cloud function executes. In Celest, you can define your own middleware and attach it to all functions in an API or to specific cloud functions.

To define your middleware, go to your <flutter_app>/celest/apis/ folder, and create a middlware.dart file (the name of the file is up to you). The following code snippet shows two middleware for logging requests and responses being defined.

import 'package:celest/celest.dart';

// A middleware that prints requests to your cloud function.
class logRequests implements Middleware {
  const logRequests();

  @override
  Handler handle(Handler handler) {
    return (request) async {
      print(request);
      return handler(request);
    };
  }
}

// A middleware that prints responses from your cloud function.
class logResponses implements Middleware {
  const logResponses();

  @override
  Handler handle(Handler handler) {
    return (request) async {
      final response = await handler(request);
      print(response);
      return response;
    };
  }
}

To attach a middleware to a cloud function, annotate the function with an instance of the middleware. The following is an example of using the request and response logging middleware in an API.

import 'package:celest/celest.dart';
import 'package:celest/api/middleware.dart' as middleware;

// Logs requests for only this function.
@middleware.logRequests()
Future<String> sayHello(
  FunctionContext context, 
  String name,
) async {
  return 'Hello, $name';
}

// Logs responses for only this function.
@middleware.logResponses()
Future<String> sayGoodbye(
  FunctionContext context, 
  String name,
) async {
  return 'Goodbye, $name';
}

You can alternatively set up middleware to run for all functions inside an API file by applying the middleware annotation at the top of your API file as shown below.

// Logs requests of every function defined in this API.
@middleware.logRequests()
library;

import 'package:celest/celest.dart';
import 'package:celest/api/middleware.dart' as middleware;

Future<String> sayHello(
  FunctionContext context, 
  String name,
) async {
  return 'Hello, $name';
}

Future<String> sayGoodbye(
  FunctionContext context, 
  String name,
) async {
  return 'Goodbye, $name';
}

You also have the option to compose middleware by applying multiple middleware to the API or cloud function. In the following example, four middleware are composed and will execute in top-down order. When a user calls sayHello, the execution order of the middleware will be: first, second, third, then fourth.

@middleware.first()
@middleware.second()
library;

import 'package:celest/celest.dart';
import 'middleware.dart' as middleware;

@middleware.third()
@middleware.fourth()
Future<String> sayHello(
  FunctionContext context, 
  String name,
) async {
  return 'Hello, $name';
}

Since middleware can apply logic before and after a function runs, the composition of the middleware can be thought of as a sandwich. That means, in the previous example, middleware.first runs both first and last if it defines both pre- and post-handler logic.

1. @middleware.first pre-handler logic runs
2. @middleware.second pre-handler logic runs
3. @middleware.third pre-handler logic runs
4. @middleware.fourth pre-handler logic runs
5. sayHello runs
6. @middleware.fourth post-handler logic runs
7. @middleware.third post-handler logic runs
8. @middleware.second post-handler logic runs
9. @middleware.first post-handler logic runs

Logging in cloud functions

Celest enables you to use logging to capture important information as your functions execute. Within your function code, you can use print() statements or a custom logger which prints to the terminal. These logs will appear in the same terminal where the celest start command is running and be accessible when running remotely post-deploy.

Here is an example of using print statements in your cloud function definition.

import 'package:celest/celest.dart';

Future<String> sayHello(
  FunctionContext context, 
  String name,
) async {
  print('Hello, $name');
  return 'Hello, $name';
}

Future<String> sayGoodbye(
  FunctionContext context, 
  String name,
) async {
  print('Goodbye, $name');
  return 'Goodbye, $name';
}

When you call the cloud function, these logs will be streamed locally.

$ celest start
Celest is running on http://localhost:7777…

Hello, Celest
Goodbye, Celest

Creating custom exceptions

You can create custom exception types in your backend to control how APIs and cloud functions behave when there are errors. This enables you to have clear exceptions thrown in your Flutter app that you can react to.

Below is an example of how to define a custom exception. You can create exceptions in any folder inside your celest folder. For this example, the exception type is defined in <flutter_app>/celest/apis/my_exception.dart.

class MyException implements Exception {
  const MyException(this.message);

  final String message;
}

You can then throw these exceptions in your functions whenever needed as shown below.

import 'package:celest/celest.dart';

import 'my_exception.dart';

Future<String> sayHello(
  FunctionContext context, 
  String name,
) async {
  // Perform custom validation
  if (name.isEmpty) {
    throw MyException('Input cannot be empty');
 }
  return 'Hello, $name';
}

In your Flutter app, the same MyException type will be thrown by the generated client if an error occurs.

import 'celest/client.dart' as celest;

Future<String> getGreeting(String name) async {
  try {
    return await celest.apis.greeting.sayHello(name);
  // Catch the exception type defined in your backend
  } on MyException catch (e) {
    print('Uh oh! Could not greet $name: $e');
    rethrow;
  }
}

Supported data types

With Celest serverless APIs and functions, serialization is handled out-of-the-box in most cases. In situations requiring custom serialization, we support any custom classes that you’re already using without any extra setup.

Imagine you're working on an e-commerce application with an Order class defined in your codebase.

class Order {
  const Order ({
    required this.id,
    required this.customerName,
    required this.price,
  });

  final int id;
  final String customerName;
  final Price price;
}

enum Currency { usd, cad, ... }

class Price {
  const Price({
    required this.currency,
    required this.dollars,
    required this.cents,
  }): assert(cents < 100);

  final Currency currency;
  final int dollars;
  final int cents;
}

You can use this Order type in any cloud function as both a parameter or return value, without the need to manually add serialization logic.

import 'package:celest/celest.dart';

import 'types/order.dart';

Future<String> createOrder(
  FunctionContext context,
  Order customerOrder,
) async {
    // ...
}

When communicating with your backend, Celest will serialize the Order class as a JSON map with the field names as keys.

{
  "id": 123,
  "customerName": "Celest",
  "price": {
    "currency": "usd",
    "dollars": 100,
    "cents": 34
  }
}

If you need custom handling over serialization logic, add a fromJson constructor and toJson method to your datatype. Celest will use your custom fromJson/toJson implementations instead when transmitting the type to and from your backend.

[!NOTE] Here, the Price.toJson method is used to upper-case the currency value.

class Price {
  // ...

  factory Price.fromJson(Map<String, dynamic> json) {
    // ...
  }

  Map<String, dynamic> toJson() => {
      'currency': currency.name.toUpperCase(),
      'dollars': dollars,
      'cents': cents,
    };
}

{
  "id": 123,
  "customerName": "Celest",
  "price": {
    "currency": "USD",
    "dollars": 100,
    "cents": 34
  }
}

Interoperability with Dart packages

Your cloud functions are pure Dart functions. They are compiled to run natively on Linux and so any Dart packages which can be used on Linux can be used in your functions.

For example, to communicate with systems outside of Celest, you can use the http package, dio, or any other HTTP package you are familiar with.

To add a package for use in your backend, run the dart pub add command from the celest folder.

$ dart pub add http

Environment variables

Environment variables can be used to provide environment-specific configuration to your backend. They allow you to keep their values separate from your codebase, improving flexibility when running in different environments.

To set up environment variables in your backend, navigate to the <flutter_app>/celest/config/env.dart file and list all the variables you’ll need throughout your backend.

import 'package:celest/celest.dart';

const EnvironmentVariable greetingUrl = EnvironmentVariable(name: 'GREETING_URL');

To ensure a cloud function has access to the variable when it runs, pass it as a parameter and annotate with the variable definition. Here, the greeting service URL will be securely injected by the server when your function starts.

[!NOTE] Annotated parameters (like greetingUrl) will not appear in the generated client, but can be used in your backend when unit testing and mocking (see Testing your backend resources below).

import 'package:celest/celest.dart';
import 'package:http/http.dart' as http;

import '../resources.dart';

Future<String> sayHello(
  FunctionContext context, 
  String name, {
  @envVariables.greetingUrl required String greetingUrl,
}) async {
  // Call an external greeting service.
  final response = await http.post(
    Uri.parse(greetingUrl).replace(path: '/sayHello'),
    body: jsonEncode({
      'name': name,
    }),
  );
  if (response.statusCode != 200) {
    throw GreetingException(
      'Failed to say hello to $name: ${response.body}',
    );
  }
  return response.body;
}

class GreetingException implements Exception {
  const GreetingException(this.message);

  final String message;
}

Setting up environment variable values locally

When you run celest start or celest deploy, the CLI will look for values of the environment variables in your shell environment. For any variables not defined, the CLI will prompt you for their values.

Please provide values for the following environment variables:
? GREETING_URL: <Enter your value>

To change the values of environment variables previously defined, re-export the value from your terminal before running celest start or celest deploy.

export GREETING_URL=<new URL> 

Celest will detect the presence of a new value and update your local/deployed function to reflect the change.

Testing your backend resources

The serverless functions and APIs you define are Dart functions and can be tested like any other. Within your celest folder, write unit tests for your functions using package:test or any other Dart testing framework.

import 'package:celest/celest.dart';
import 'package:test/test.dart';

import '../apis/greeting.dart';

void main() {
  test('sayHello', () async {
    final result = await sayHello(
      FunctionContext.test(),
      'Celest',
      greetingUrl: 'http://localhost:8000',
    );
    expect(result, 'Hello, Celest');
  });
}

Deploying your backend resources

When you have tested and validated your backend locally, use the Celest CLI to deploy your backend resources to the cloud.

$ celest deploy

Calling Celest APIs with HTTP requests

If you'd like to use your Celest APIs outside of your Flutter/Dart app, you still can! Celest functions are exposed as HTTP endpoints which can be called via any HTTP client from any programming language or toolchain.

The HTTP conventions for Celest functions are:

• JSON requests/responses
• POST requests
• 200 status code for success
• 400 status code for exceptions
• 500 status code for errors

When a cloud function fails with an exception or error, the response will carry a 4xx/5xx status code and JSON body with an error key. If the exception is a user-defined exception type, the error field itself is encoded as a JSON message.

For example, if a function throws the MyException type defined in the example above, the response would be:

400 Bad Request
{
  "error": {
    "$type": "MyException",
    "message": "Input cannot be empty"
  }
}

However, if the function threw a StateError, it would look like this where the error is stringified in the error field.

500 Internal Server Error
{
  "error": "Bad state: Something bad happened"
}

Next steps

Thank you for your time and for making it this far! We greatly appreciate any feedback you provide on our developer experience.

If there’s anything we’ve left out or things you would like to see discussed, please us know! Shortly, we will share additional RFCs to cover:

• WebSockets
• Defining multiple backend environments
• Controlling access to your APIs
• Event-driven patterns such as scheduled functions

Thank you for coming on this journey with us 💙 We are so excited to bring you these features and more very soon! 🚀

If you haven’t already, please make sure to sign up for our waitlist to get the latest updates on our progress and follow us on Twitter/X where we share more insights and behind-the-scenes snippets.

[caseycrogers]

Can I hit my Celest functions via REST requests?

For things like your Dart native server errors (very cool btw), what do those look like when I hit a Celest API via REST and it errors? Maybe there could be a way to specify the HTTP status code of a given exception (or success for that matter)?

Context: we have a mess of Python scripts that we're trying to run on a schedule. I'd be interesting in migrating them to Celest and then triggering them via HTTP request (though I imagine longterm Celest will support triggering functions on a schedule out of the box?)

[dnys1]

Update: I've added a new section to the RFC titled Calling Celest APIs with HTTP requests which summarizes these points.

---

Hey Casey! Great questions. Here's what we're thinking on that front.

Can I hit my Celest functions via REST requests?

Yes! Celest functions are exposed as HTTP endpoints which can be called via any HTTP client from any programming language or toolchain.

The HTTP conventions for Celest functions are:

• JSON requests/responses
• POST requests
• 200 status code for success
• 400 status code for exceptions
• 500 status code for errors

For things like your Dart native server errors (very cool btw), what do those look like when I hit a Celest API via REST and it errors?

When a cloud function fails with an exception or error, the response will carry a 4xx/5xx status code and JSON body with an error key. If the exception is a user-defined exception type, the error field itself is encoded as a JSON message.

For example, if a function throws the MyException type defined in the example, the response would be:

400 Bad Request
{
  "error": {
    "$type": "MyException",
    "message": "Input cannot be empty"
  }
}

However, if the function threw a StateError, for example, it would look like this where the error is stringified in the error field.

500 Internal Server Error
{
  "error": "Bad state: Something bad happened"
}

Maybe there could be a way to specify the HTTP status code of a given exception (or success for that matter)?

We have considered this and we have decided for our initial release not to allow configuration of HTTP abstractions in Celest APIs including response codes. We think that fully leveraging Dart's idioms provides the best developer experience and that exposing HTTP/REST semantics may take away from that goal.

But we would love to hear if you think there is still an opportunity to improve the offering with status codes or other configurations given the examples above.

Context: we have a mess of Python scripts that we're trying to run on a schedule. I'd be interesting in migrating them to Celest and then triggering them via HTTP request (though I imagine longterm Celest will support triggering functions on a schedule out of the box?)

You're absolutely correct that we intend to introduce scheduled functions and other event patterns natively in Celest. For the time being, it sounds like you'll be able to use deployed Celest functions in your python scripts, though please let me know if it seems like there would be any sticking points with that integration.

Thanks for the awesome feedback!

[caseycrogers]

Np, and your conclusion-to hold off on any more complex status code features-seems pretty reasonable.

The "as if it were a local function" level of experience you're working towards seems like an ideal selling point-and I could still do my goofy REST stuff off to the side while I wait for more native cron scheduling features. On that note, a longterm feature I'd love to see: An annotation that is something like: @Schedule(Duration). It'd take either a duration or a date time and trigger the function every Duration units of time (and/or more complex date oriented scheduling like "first of every month" or "every Tuesday"...)

A separate thought: With server code feeling as if it were client code, it seems confusing to manage server<>client compatibility and ensure that new server code ships with new client code. How are you managing deploys and are there any tools to make this easier?

To give concrete examples:

1. I ship v1
2. I make a breaking change and ship v2. v2 clients will crash trying to talk to v1 server functions, and vice versa How do I ensure that residual v1 clients continue to hit server v1 and new v2 clients hit server v2?

Oh crap. v2 has a bug in one of its server functions. I want to rush out a hotfix without having to wait for a new client to go out. How do I now deploy a new version of my functions and ensure my previous client hits the new functions? And then, how do I avoid server versioning and client versioning diverging to the point where it's really hard to remember the associations between the two?

[abdallahshaban557]

An annotation that is something like: @Schedule(Duration). It'd take either a duration or a date time and trigger the function every Duration units of time (and/or more complex date oriented scheduling like "first of every month" or "every Tuesday"...)

Hi @caseycrogers - we are thinking of a pretty similar pattern for scheduling cloud functions. We want to make sure we are doing it in a way that would be consistent across different parts of the features we introduce. That is a topic in one of our future RFC around event-driven patterns.

With server code feeling as if it were client code, it seems confusing to manage server<>client compatibility and ensure that new server code ships with new client code.

For adding changes that would be considered breaking to your existing apps, we are still thinking about the best pattern to adopt to provide a path forward for developers iterating on their deployed backend. We will share our thoughts in future RFCs.

[filippomenchini]

Hi @dnys1 , @abdallahshaban557 !

I have a couple of questions:

• How are you doing the serialization/deserialization of classes? If I'm understanding correctly, you don't need to define the classic fromJson and toJson methods. Is it possible to know how are you managing this?
• Would it be possible to have a "celest-only" repository without the frontend? This would be useful when you have to share the backend with multiple projects, you could export the generated clients for Dart applications to use, while other languages could communicate with the backend via REST calls.
• It would be interesting to use gRPC, have you thought of that?

Keep up the good work, I'm really hyped for this 🚀

[dnys1]

Hey @filippomenchini 👋 thanks for your comment!

How are you doing the serialization/deserialization of classes? If I'm understanding correctly, you don't need to define the classic fromJson and toJson methods. Is it possible to know how are you managing this?

The CLI creates serializers for your classes in a very similar way to how packages like json_serializable and built_value work. We examine the structure of your classes and types and create the necessary functions to handle serialization for you. While this provides a convenient and stable out-of-the-box solution, it is opinionated which is why we offer the ability to define your own fromJson/toJson methods if our specification doesn't meet your needs.

Would it be possible to have a "celest-only" repository without the frontend? This would be useful when you have to share the backend with multiple projects, you could export the generated clients for Dart applications to use, while other languages could communicate with the backend via REST calls.

Yes! We are exploring the best way to enable this pattern for developers and will be happy to share more specifics in an upcoming RFC.

To better understand your setup, are you envisioning that your Flutter/Dart clients would be in wholly separate Git repos from the Celest backend? If so, would it be a dealbreaker if Celest only supported monorepos where your client + backend code lived in a single repository?

It would be interesting to use gRPC, have you thought of that?

Yes, we are huge fans of gRPC and actually make use of it in our backend! We chose not to implement Celest APIs on top of gRPC for a few reasons:

• Debugging is generally harder than with REST/JSON
• Not all gRPC features are supported on the web (specifically when it comes to streaming)
• Managing an intermediate layer of protobufs ran counter to our approach of everything in Dart

I can say that we hope to bring all the best features of gRPC to Celest but expressed via Dart instead of Protobuf!

[filippomenchini]

Thanks for the quick response!

While this provides a convenient and stable out-of-the-box solution, it is opinionated which is why we offer the ability to define your own fromJson/toJson methods if our specification doesn't meet your needs.

Got it, sounds great!

To better understand your setup, are you envisioning that your Flutter/Dart clients would be in wholly separate Git repos from the Celest backend? If so, would it be a dealbreaker if Celest only supported monorepos where your client + backend code lived in a single repository?

Yes, I was thinking about many Flutter/Dart clients linked to a single backend that is managed in a independent repo, this could be useful in some situations, but it wouldn't be a dealbreaker if Celest only supported monorepos, you could always have a folder with the Celest code and many others containing the applications. With Celest this should be pretty easy to maintain, unlike AWS Amplify or Firebase that required you to be able to switch programming languages every 2 seconds. Monorepos also facilitate versioning and deployment, this is a big plus for me! Also, automated testing should be really sweet with Celest, especially integration tests.

Yes, we are huge fans of gRPC and actually make use of it in our backend! We chose not to implement Celest APIs on top of gRPC for a few reasons: Debugging is generally harder than with REST/JSON Not all gRPC features are supported on the web (specifically when it comes to streaming) Managing an intermediate layer of protobufs ran counter to our approach of everything in Dart I can say that we hope to bring all the best features of gRPC to Celest but expressed via Dart instead of Protobuf!

This makes sense, maybe you could tackle this in the future, REST APIs are more accessible and much easier to implement!

[dnys1]

Yes, I was thinking about many Flutter/Dart clients linked to a single backend that is managed in a independent repo, this could be useful in some situations, but it wouldn't be a dealbreaker if Celest only supported monorepos, you could always have a folder with the Celest code and many others containing the applications.

Thanks for sharing more about your envisioned repo setup! For Celest, we are really excited about the experience we can provide in monorepos, including a simple and intuitive CI/CD experience. That being said, we're not closing the door to other ways people build their apps. We'll definitely keep your feedback in mind as we build and get your insights when we've fleshed out more.

This makes sense, maybe you could tackle this in the future, REST APIs are more accessible and much easier to implement!

We'll keep this in mind too. Part of our vision with Celest is to simply find the right abstractions for RPC-style communication. I think if we can nail that, then the door is wide open for extending support to other communication patterns like gRPC 🚀

[filippomenchini]

That being said, we're not closing the door to other ways people build their apps. We'll definitely keep your feedback in mind as we build and get your insights when we've fleshed out more.

When Celest will be released, I'll be happy to try the multi-repos setup to give you a feedback 🚀

[lukepighetti]

1. Scheduling annotation is huge imo
2. Any chance for a distributed task dispatcher like celery?
3. What about a distributed cache? Laravel has a really cool story around this
4. Are you designing with stateful connections like websockets in mind? (You can do Future now, but what about Stream (server side streaming)? Or an argument that is Stream (for client side streaming)

[abdallahshaban557]

Hi @lukepighetti

Scheduling annotation is huge imo

We feel the same. Top of mind for us for sure. The experience that Casey mentioned earlier in this RFC is pretty similar to what we have so far.

Any chance for a distributed task dispatcher like celery?

We are thinking about queuing and creating abstractions for enabling event-driven applications. Would love to understand more which use cases you are hoping it unlocks for you.

What about a distributed cache? Laravel has a really cool story around this

We haven't thought deeply about a distributed cache per se, but we have been chatting about offline first for Flutter apps. Were you hoping for caching to solve for on-device caching? or something else?

Are you designing with stateful connections like websockets in mind?

Websockets are going to be in a future RFC that we will share. We have some initial ideas but wanted to spend a bit more time fleshing it out and walking through more scenarios. We want to get the right experience down including Pub/Sub patterns and applying authorization.

[chimon2000]

When you run celest start or celest deploy, the CLI will look for values of the environment variables in your shell environment. For any variables not defined, the CLI will prompt you for their values.

It might be useful to have a celest env command to get/set environment variables that would be read from a .env file. I am imagining it working similar to Vercel.

[abdallahshaban557]

Hello @chimon2000

So you would prefer to use the .env file rather than exporting out individual environment variables in your terminal? Do you typically use .env files with your Flutter apps? If so, we'd love to hear more about how you manage and share these .env files with your team members.

[marcglasberg]

1. Are functions guaranteed to run once and only once? Or is it at least once? In the latter case they need to be idempotent, which is a pain. If they fail to connect (device has no internet, or it has but the backend is down, or other network error) and hasn't reached the backend, do we get a special exception thrown by the function, in the client? Or will it retry automatically with exponential backoff a certain number of times, and only then fail? Is that configurable?

2. Are functions reused? In which case, just like Firebase and AWS Lamba, you can have a kind of best-effort small local cache with globals or a map object in the FunctionContext. Anyway, not very important.

3. Can you provide more info about FunctionContext? What does it contain? What does FunctionContext.test() do? Is .test() that a constructor that allows you build a specific context for testing?

4. It's great that you accept toJson and fromJson to serialize. My package https://pub.dev/packages/fast_immutable_collections already contains toJson() and fromJson() for its collections. Just to confirm, if I have IList([Student('John'), Student('Eva')]).toJson(), but I have not defined toJson() for the Student class, can my IList.toJson() method return a regular list of students: [Student('John'), Student('Eva')] and you will accept this and serialize the Student objects yourself?

5. If one day you create local caching to allow for offline first, I'd suggest it should be explicit (the client should ask for it, and configure it). Firebase integrates too deeply the cache with its reads, and this creates all sort of problems. Also, if you do local cache you will need to save stuff to do local disk, so that you don't loose information if the app is killed before it has a chance to connect to synchronize. I think a local cache would probably be one of the hardest things to get right.

6. Maybe you could provide plugins for IntelliJ and VS Code, to do simple stuff like installing the CLI, deploying etc.

7. I have a suggestion on how to deal with the changing of the serialized objects, as I've dealt with this many times and have a preferred way to solve this. I'll write something out and post here.

   How do I ensure that residual v1 clients continue to hit server v1 and new v2 clients hit server v2?

8. Regarding your listed next steps, I think auth is the one I'd like to know about the most. And that's maybe the only feature you could provide an optional default (but configurable) UI for, as it's both difficult to implement and almost always the same for all apps.

[dishankjindal1]

Hi @dnys1 @abdallahshaban557 - Great going so far.

Can you provide some examples for implementing authentication based api? And then only authorised users are able to trigger example greetings api?

[abdallahshaban557]

@dishankjindal1 - yes! We will provide that in a future RFC. Since we won't have an Authentication solution so far, what do you use as your primary Authentication provider that you would want to bring in and use with Celest?

@marcglasberg - Dillon and I are discussing some of your questions. Will get back to you ASAP.

[dishankjindal1]

Hey @abdallahshaban557 - I am assuming, all 0Auth 2.0, OIDC based service providers work the same. So if you can create an generic abstraction class in celest and developers can use that abstract to plug in there service providers callbacks, api tokens, etc. Hence covering all services providers will be a great addition.

In my workplace, I have used both Auth0 and AWS cognito based implementations. But because I am on the frontend I am not sure how they are managing the resources.

I just need few APIs,

• login_api,
• logout_api,
• refresh_token_api,

For state management,

• user.authencticated (Boolean) (highly reactive) need it from the beginning, just before calling the runApp

Cc: @dnys1

[dnys1]

@marcglasberg Thanks for your awesome feedback!

1. Are functions guaranteed to run once and only once? Or is it at least once? In the latter case they need to be idempotent, which is a pain. If they fail to connect (device has no internet, or it has but the backend is down, or other network error) and hasn't reached the backend, do we get a special exception thrown by the function, in the client? Or will it retry automatically with exponential backoff a certain number of times, and only then fail? Is that configurable?

Great question. For retry, we plan to handle that automatically for developers. Transient errors will be exposed in such a way that they can be caught and handled by developers. For idempotency, what would you think about an @api.idempotent() annotation which exposed an idemopotencyKey on the client? We would handle the caching logic on the server for you.

2. Are functions reused? In which case, just like Firebase and AWS Lamba, you can have a kind of best-effort small local cache with globals or a map object in the FunctionContext. Anyway, not very important.

Functions are reused. That might change in the future and we will be treating it as an implementation detail. We recommend developers to not use global state for caching in their functions.

3. Can you provide more info about FunctionContext? What does it contain? What does FunctionContext.test() do? Is .test() that a constructor that allows you build a specific context for testing?

FunctionContext is a parameter that allows you to access specific information around the Cloud Function execution. It will include information such as IP address of calling client, and in the future any additional utilities developers would need from us.

4. It's great that you accept toJson and fromJson to serialize. My package https://pub.dev/packages/fast_immutable_collections already contains toJson() and fromJson() for its collections. Just to confirm, if I have IList([Student('John'), Student('Eva')]).toJson(), but I have not defined toJson() for the Student class, can my IList.toJson() method return a regular list of students: [Student('John'), Student('Eva')] and you will accept this and serialize the Student objects yourself?

We’re basically using json_serializable under the hood and your package is compatible with that. I checked and it will work out of the box for homongeous collections, e.g. IList<Student> or IMap<String, Student> even if Student.toJson/Student.fromJson are not defined.

5. If one day you create local caching to allow for offline first, I'd suggest it should be explicit (the client should ask for it, and configure it). Firebase integrates too deeply the cache with its reads, and this creates all sort of problems. Also, if you do local cache you will need to save stuff to do local disk, so that you don't loose information if the app is killed before it has a chance to connect to synchronize. I think a local cache would probably be one of the hardest things to get right.

Abdallah and I worked on AWS Amplify Flutter where we had an offline first solution - and we agree with that sentiment. Developers have to opt in to it. Ideally, we want to allow caching/offline first even at the model level to provide the best level of control for developers. Agreed as well that this will be one of the more difficult problem to solve for, but we’re excited for the challenge.

6. Maybe you could provide plugins for IntelliJ and VS Code, to do simple stuff like installing the CLI, deploying etc.

We are exploring that! A dev tools extension also seem like a really cool alternative. Still brainstorming at this point so we appreciate the feedback.

7. I have a suggestion on how to deal with the changing of the serialized objects, as I've dealt with this many times and have a preferred way to solve this. I'll write something out and post here.

How do I ensure that residual v1 clients continue to hit server v1 and new v2 clients hit server v2?

That sounds great! Looking forward to discussing it more. Also feel free to reach out if you would like to schedule a call to go over it in more depth.

8. Regarding your listed next steps, I think auth is the one I'd like to know about the most. And that's maybe the only feature you could provide an optional default (but configurable) UI for, as it's both difficult to implement and almost always the same for all apps.

For Auth, our thought process is that we will offer developers magic links, WebAuthN, and social sign-in. You will also be able to integrate any custom OIDC-compliant identity provider. During our time at AWS Amplify, we built the Authenticator UI component which took care of all the UI flows, so we understand first-hand the value of offering that for developers.

[abdallahshaban557]

Hi @dishankjindal1 - We are thinking very similarly in terms of integrating with OAuth2 and OIDC-compliant providers. This piece will be in one of the future RFCs we share since that is how we expect customers will control access to their APIs in the absence of Celest having an Authentication solution in the short term.

[chimon2000]

Hello @chimon2000

So you would prefer to use the .env file rather than exporting out individual environment variables in your terminal? Do you typically use .env files with your Flutter apps? If so, we'd love to hear more about how you manage and share these .env files with your team members.

Yes, in our apps normally we use a .env configuration file rather than Dart Defines for environment variables. Specific to Flutter we use a codegen library to inject variables into the application as part of a build step.

[chimon2000]

Abdallah and I worked on AWS Amplify Flutter where we had an offline first solution - and we agree with that sentiment. Developers have to opt in to it. Ideally, we want to allow caching/offline first even at the model level to provide the best level of control for developers. Agreed as well that this will be one of the more difficult problem to solve for, but we’re excited for the challenge.

It'd be nice if the offline first solution were pluggable. Powersync for example integrates with any Postgres database and the client essentially just uses their SQLite library.

[abdallahshaban557]

Hi @chimon2000 , the environment variables we want to help you manage are all related to information you want stored and used only in your backend. We have been revising our experience, and here is what we landed on so far.

Setting environment variables:

• If a developer wants to set up environment variables, they would have to use the celest env set command.
• When they use it, developer would then be prompted to enter the environment they are making changes to. Which would also include their local environment.
• Once they do that, we can then prompt them to:
  • Add environment variable
    • They will have to provide the key and value for that variable.
  • Upload list of env variables
    • We will instruct developers to drop their .env files in the <flutter_app>/celest/config/ folder, and we will show a list of the files from the CLI that a developer can use to mass upload/update their env variables. We think this will be easier than having to provide a path to the .env files.
  • Delete env variable
    • Developers are presented with a list of env variables that they can then delete.

After setting the environment variables, the CLI will take care of generating the types needed for you to inject the environment variables into your cloud functions.

Getting environment variables:

• At any point, a developer can ask to get ALL values defined in an environment by using the celest env download command.
  • First, a developer will have to celest the environment they want to download the environment variables for.
  • The developer picks the environment, and then the env file is downloaded as <env_name>.env inside the config folder.
    • If there is a conflict in the name, the developer will have to confirm that this command will override their current existing <env_name>.dart file on disk.

Please let us know what you think - we've taken inspiration from the Vercel link you sent out.

On your point about keeping the offline logic pluggable, we've taken note of that from what last time we've chatted. We will for sure keep it in mind!

[marcglasberg]

@dnys1 @abdallahshaban557 Do you mean idempotency would work like described here? https://stripe.com/docs/api/idempotent_requests

[abdallahshaban557]

Hi @marcglasberg - yes, that is exactly what we mean!

[marcglasberg]

A question:

@api.anonymous()
library;

import 'package:celest/api.dart' as api;

Future<String> sayHello(
  FunctionContext context,
  String name,
) async {
  return 'Hello, $name';
}

Future<String> sayGoodbye(
  FunctionContext context,
  String name,
) async {
  return 'Goodbye, $name';
}

I see you don't need to annotate the functions to turn them into cloud functions. But what if I want a cloud function to call a local function?

@api.anonymous()
library;

import 'package:celest/api.dart' as api;

Future<String> sayHello(
  FunctionContext context,
  String name,
) async {
  return composePhrase(name);
}

String composePhrase(String name) {
  return 'Hello, $name';
}

Am I correct to assume this wouldn't work, as it would try to turn composePhrase into a cloud function (and fail as it's not the correct signature)?

Or will it actually work, by only turning into cloud functions those containing FunctionContext context in the signature?

[dnys1]

Hi Marcelo, good observation! To define local functions which are not cloud functions, there are two options available:

1. Define them as private functions (e.g. change to _composePhrase)
2. Define them outside the apis folder and import them into the API file.

Any top-level, non-private functions within the apis folder are considered cloud functions.