Restler 3.0 Feature Requests and Ideas

[electrical]

Since you asked for idea's, let me start with some :-)

1. Output structure: http://groups.google.com/group/api-craft/browse_thread/thread/9b046e863b5943df. Metadata is something that can be generated from the API it self. Links can be generated from the class the user builds.
2. Correct header settings: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html. Think some things here can be found that are useful. Like: 14.7 Allow
3. Pagination: limiting the returning result set. Can be build in the api class of the user self, but perhaps some core functionality can be build in the API core ?
4. JsonP: JsonP as part of the package; JsonP becomes more and more important with cross-domain usages.
5. Versioning Support: To have an inbuilt Restler way to manage your API versions (Request from @KitCarrau in issue #21)

More will follow when they pop up in my mind :-)

[Luracast]

Hi @electrical, Thanks for the initiative and suggestions, lets use this issue page for all our discussions :)

[electrical]

For item 5 i do have my questions with that. Versioning of the backend ( classes ) may go very quickly but the interface it self ( the urls ) don't. Also if you read http://blog.apigee.com/detail/restful_api_design_tips_for_versioning/ i think the versioning question is very clear.

Unless im missing a part of the versioning idea brought by @KitCarrau

Btw. i think allot of good idea's and practical solutions can be found here: http://apigee.com/about/resources

[roynasser]

I have implemented simple versioning including a default version and option for vXXX in my url as the first parameter. I believe i submitted it in the other issue I was commenting with luracast... Versioning is extremely important as new features are added, and things eventually need to be changed around...

Anyways, I have one suggestion for V3. a way of "piling on" requests... basically, instead of having my ajax client have to refresh 3 different areas of the page in 3 different requests, they could call a specific request and either have an "array" of commands to execute (might be a bit too tricky), or maybe a specific route which includes N functions from more than one class even?

This would be very useful in saving authentication calls, DB calls, and in saving network latency as a whole...

class-based authentication awareness will need to be visited for this to work correctly, and a few other things must be taken care of, but overall it would be an interesting thing to tackle I think... The return itself could be as 3 combined objects, references by the function name or something of the sort? Intersting stuff! :)

The idea for this isnt really to combine all the calls you would ever make (if you are using the API to drive a site, not assuming you are using it as a public developer api), but to combine keepalives from browsers, etc... Another way is to just plan all the possible combinations of functions you need to call together and encapsulate them manually under a new class... (but it wouldnt be as elegant :p)

[electrical]

As i stated earlier, allot of stuff at Apigee are extremely useful;

Limiting result sets and selecting return fields: http://blog.apigee.com/detail/restful_api_design_can_your_api_give_developers_just_the_information/

Output format: http://blog.apigee.com/detail/restful_api_design_support_multiple_formats/ ( think you already have this in a good way; the 2 options i mean )

Returning error codes: http://blog.apigee.com/detail/restful_api_design_what_about_errors/ ( which you already have fully implemented i believe ) Additional what i believe is important is the ability to suppress the error code returning. Instead of sending a ( for example ) 5xx code, always send a 200 code with the error code in the body with the description. ( this is for systems that don't understand http error codes or can't handle it )

See http://www.slideshare.net/apigee/restful-api-design-second-edition for more idea's.

[electrical]

an other thing is logging. i think it could be useful to have some kind of audit trace of all calls that have been made. Things like the IP address, date and url called are important. If some one wants a real verbose log you could also include the input and output data.

[electrical]

Perhaps it's possible to create hooks inside the class to provide the possibility to create custom classes to hook into the application. This way things like a logging class can easy be created. Also things like Content checking and other verification things can be created.

It would require a before_something and after_something hooks. The functions need to be registered trough a $resther->RegisterHook("class","function","hookname"); function. The return value should be a true or false.

Just an idea :-)

[robderijk]

Hi,

I hope this is within the scope of this project but it would be nice if there is a possibility for (automatically) creating API documentation. Things like authentication, which HTTP methods (GET, POST..) are supported for each URL, what the URL's are etc. I was inspired by http://swagger.wordnik.com/ I like to hear your ideas about it.

[Luracast]

Here is what we are thinking,

Every version of Restler has a theme behind it

Restler 1: Don't reinvent the wheel

Leverage existing knowledge, defined methods. Reuse existing code

Restler 2: Convention over configuration

Write less code to get most of it

Restler 3: Best Practices

Both for PHP and API

Some of the changes we are considering are

• Reduce creation of multi-level routes to reduce the ambiguity. Avoid mapping optional parameters to url. They will be swept under ? query param
• Improved metadata support to enable API discovery through possible ways like WADL, Swagger UI, Google API discovery etc (@murdock - hope this answers your question)
• Reference implementation for pragmatic rest (as per the apigee recommendation) Version support in the url using spl_autoload. If we declare $restler->apiVersion = 3 then we will prepend /v3/ to the URL and try to look for the API class in /v3/ folder. If not found we will continue looking through/v2/ and /v1/ until we find the class and include. This makes it possible that we can upgrade the classes and put the class files in relevant folder only when we make some api change. Users can use versions from 1 to 3 where less than 3 will be marked deprecated.
• Improve documentation to include
  • more examples
  • flow diagrams
  • installation on
  • Apache
  • Nginx
  • Cloud hosting platforms
• Caching support
• Logging support
• Setting custom positive response code through phpdoc comment
• Versioning through header (proper REST way)
• Reducing the boilerplate code by implementing input validation through phpdoc comments

[Luracast]

Here is the result of one of our experiments. following code produces the documentation shown in the picture below

<?php
class Authors
{
    public $dp;
    static $FIELDS = array('name', 'email');
    function __construct ()
    {
        /**
         * $this->dp = new DB_Session();
         * $this->dp = new DB_PDO_MySQL();
         * $this->dp = new DB_Serialized_File();
         */
        $this->dp = new DB_PDO_Sqlite();
    }

    /**
     * Get Author for the specified Author ID
     * 
     * Passing a valid author id, will result in getting 
     * details (name, email, author id) of the specific
     * Author.
     * @param int $id {min=1&max=2&error=AuthorID+is+out+of+range} AuthorID
     * @throws RestException 417 one or more of required fieds missing
     * @throws RestException 404 Author not found
     * @return Author author instance for the given id;
     */
    function get ($id)
    {
        return is_null($id) ? $this->dp->getAll() : $this->dp->get($id);
    }

    /**
     * Create new Author
     * @param Array $request_data
     * @throws RestException 417 when one or more of required fieds missing
     */
    function post ($request_data)
    {
        return $this->dp->insert($this->_validate($request_data));
    }

    /**
     * Update Author information
     * @param int $id
     * @param array $request_data
     */
    function put ($id, $request_data)
    {
        return $this->dp->update($id, $this->_validate($request_data));
    }

    /**
     * Delete Author by ID
     * @param int $id
     */
    function delete ($id)
    {
        return $this->dp->delete($id);
    }

    private function _validate ($data)
    {
        $author = array();
        foreach (self::$FIELDS as $field) {
            //you may also vaildate the data here
            if (! isset($data[$field]))
                throw new RestException(417, "$field field missing");
            $author[$field] = $data[$field];
        }
        return $author;
    }
}

[robderijk]

I get it, thanks for the explanation!

[electrical]

Very cool for the automated documentation :-)

I'm not sure about the versioning trough header definitions. Although it might be pure REST way, it's not recognized as a best practice way. One could do it of course 2 ways;

1. The headers as you mentioned and have 1 gateway url for all API versions.
2. Have a separate URL for each API version.

just my 2 cents :-)

[electrical]

Btw, for that before and after hook proposal i wrote earlier; you can use it for different things. Logging, Rate limiting, content validation ( like creating an md5 sum to avoid content tempering ), etc.

[kzap]

I like how Restler is so simple to implement in terms of classes and code. I like how it get expose methods in Classes. One way I see in using this is to use your existing classes as a base for the API so you can keep your code organized rather than having to duplicate code in 2 locations.

Towards this instead of the normal way where all methods except those starting with an underscore are exposed, have an alternate option where all methods starting with a configured Prefix will be exposed. That way one could just call existing classes in restler without having to change existing methods.

[Luracast]

Using Hooks

@electrical I like your hooks idea and planning to add another set of classes called Manager Classes. They can be added using

<?php
$r->addManagerClass('RateLimitManager');

These classes can have predefined methods that will be called automatically (something like a event listener) when respective method is being executed in restler

Does that make sense?

Code Reuse

@kzap thanks for the complements, they keep us motivated :)

I too was thinking the same when I was working on Restler 2.0 specs. When we prefer convention over configuration we get to loose something. In Restler 2.0 reusing existing code base with out modification was lost.

I'm thinking about adding two more url directives in Restler 3.0 using PHPDoc comments to fix this

They are

• @url+ GET /custom/url/route to include more url mapping to the urls added by convention
• @url- or @url- GET /remove/only/this/route this can even be defined at class level to disable auto routing for a specific class

What do you think?

[roynasser]

I really like the documentation you gave as example, easy to add as coding and quite informative... the param comment for data validation still isnt valid for restler 2, right? Would that also go into the documentation (shoudlnt the doc show that the value for author id should be between 1 and 2 as specified, perhaps even show the error so that an application can be coded to handle such aPI error? but I guess this is specific to the documentation generator?)?

Is there a reference for which phpdoc style config options are valid today? (restler 2)? and a roadmap for additional ones for 3 you can maybe post?

Any thoughts or ideas for multi-function return? (short of doing an array_push from multiple functions and manual validation instead of the built-in restler class authentication)? (see my comment above for "combining mutliple calls")...

My 2 cents on the versioning is that having it in the URL would also be useful in some circumstances...

We use the API to serve our own internal system, so some parts of the API are "public" and can be accessed by developers, while others are merely furnishing data to our UI. For this we use Ajax calls to pull in json objects that interface to ext or similar framework to populate our own UI with data, generate grahs, etc...

We found restler to be very efficient for this!

[Luracast]

API discovery & Documentation

@RVN-BR Swagger Spec currently has allowableValues and allowMultiple for specifying them, but not used in the UI yet. Since they are still evolving and they seems to be open to suggestions, we can improve it over time. We can currently use the description to show it to the api user.

PHPDoc options that are used in Restler 2.0

All the phpdoc comment elements are parsed and kept as metadata but only the following is used by the framework

• @url {GET | POST | PUT | DELETE | OPTIONS} /mapped/route allows multiple
• @class {className} (param=value&p2=v2) allows multiple to target many classes (one per class). used for setting values for the properties of specified class such as setting the root name for xml formatter.

Multi-function call per one api call

I'm still thinking on how make it work in an elegant way, and should I handle it at restler level or leave it to the api class will keep you posted

[roynasser]

Great! Thanks for the response...

1 question, sorry for jacking, we can erase this after, @class is 1 per {className}, correct? with as many params as necessary (param1=a&param2=b&paramN=ZZ), right? I tried setting it in a multilevel way for authentication as such, adding in different areas:

so (not exact code, some stuff stripped, but just to get an idea)

<?php
/**
* @class Authentication (ClassName=MyClass)
*/
class MyClass {
/**
* @class Authentication (ModuleName=MyFunction)  //// This piece never seems to get passed through...
*/
function MyFunction(){
}
/**
* @class Authentication (ModuleName=MyFunctionOther)  //// This piece never seems to get passed through...
*/
function MyFunctionOther(){
}

}

seems not to work... while below does...

<?php
class MyClass {
/**
* @class Authentication (ClassName=MyClass&ModuleName=MyFunction)
*/
function MyFunction(){

}
/**
* @class Authentication (ClassName=MyClass&ModuleName=MyFunctionOther)
*/
function MyFunctionOther(){

}

}

would this be something that could maybe be made more flexible? it would make the code within slightly more portable, and easier to change some things such as Class names, etc, as well as preventing repeating the same class name every time...

Another thing would be possiblity of having a url match POST/PUT, or GET/POST? I realize today we can add as many routes as we want using the @url, but maybe combining them would make it shorter... (I do realize it is probably unwise as GET is for viewing, POST is for writing, etc, etc)... depends on how "flexible" to the developer restler aims to be... in terms of being "spec-strict" this last suggestion is probably counter-intuitive...

[Luracast]

@RVN-BR I've updated my answer above to answer your question. I will look at ways of improving it

One concern is to keep the doc parsing light, because we do that every time when we run on debug mode, which is the default. Only when we turn on the production mode as shown below, we will use the cached route stored in routes.php

also having too many ways to do the same thing is more confusing and difficult to document

<?php

$r = new Restler(TRUE); // run in production mode
# $r->refreshCache(); // call this if you need to rebuild the routes in production mode
...

first time when you run in production mode or when the routes.php file is not found it is created and holds the routing information future requests will be served by including this file for routing information. These routes wont change if you modify your api class files

if you need to rebuild the routes you can delete the routes.php or call refreshCache as shown above

Also note that writing routes.php requires proper file permissions set in the server, otherwise it will keep generating the route for every call

[electrical]

Hi guys, sorry for my late reply. I'm currently on vacation :-)

@Luracast The hook method you described makes sense yeah. All information regarding the request ( IP, method, data, etc ) will need to be parsed to it.

I think the mainframe class that i supplied to you some time ago will help with this to keep it all sane ?

For the url mapping with the @url+ and @url- im not sure if that's useful. In general, if one sets it up properly the correct paths will be auto created. Especially with defining the root of the class with addAPIClass("class","root/url") i'ts pretty easy.

In some classes i have for example a root specified as "system/password" In the class i have paths defined as @url POST /settings for a certain function. So the path would be then system/password/settings

Can't get easier then that i think :-)

@RVN-BR The url matching is something i really would avoid. Programming wise you make separate functions for each type of call which makes sense. combining them will only make the code ugly but also for the person using it makes it confusing.

The multi function call per 1 API call is something very tricky because you need some how to specify all the functions and their input data and methods. At this moment i don't see any good REST way to do that.

[Luracast]

Manager Classes

All information regarding the request ( IP, method, data, etc ) will need to be passed to it.

@electrical restler wont be passing them individually, we will set restler property of the manager class with the restler instance that is currently running. then we have the entire wealth of information and also restler to tweak. I'm not afraid to make restler a decency here as manager class is specific to Restler

Route Customization

I believe @url+ and @url- will be useful in some special cases for adding to and subtracting from the auto routes

• if we add just @url- at method level auto routing will be disabled for that method or if it is specified at the class level, it will disable auto routing for the entire class. This does not remove manual routes specified using doc comments
• we can use something like @url- GET specific/route to remove specific auto route
• where as @url+ should always follow with the specific route to be added to the generated auto routes

All these are needed because once we add a @url to a method we stop generating auto routes for that method. Only way to keep adding or removing from autoroutes is to avoid using @url and use @url+ or @url- instead.

Multi-function call

Unlike @RVN-BR, the way I'm thinking about multifunction call is for relationships something like the following

GET owners -- gets all the owners, can be filtered using query parameters
GET owners/{id} -- gets a specific owner
GET owners/{id}/dogs - gets all the dogs owned by the owner with the specific id
GET owners/{id}/dogs/{id} - gets specific dog owned by specific owner

Here if we can have some way to set the relationships and pass the result of owners/{id} from owners class query to dogs class for filtering that might simplify the code, I'm still thinking!

[kzap]

@Luracast Yes i think PHP Docs to remove or disable entire routes would be good so once could customize the route. Though this might be bypassed somehow by direct queries? I was thinking a switch to only allow routes which are specified in the PHPDocs and no default routes?

[Luracast]

@kzap updated the above to answer your question. Hope I made it clear and it makes sense as a solution.

[Luracast]

Route Customization - Further thoughts

After implementing @url- the way I described above, I'm looking at the bulkiness of the code and thinking about the confusions it can create.

Now I'm thinking I should simply remove @url+ totally and keep @url- with out the specific routes option.

We can come up with a better name for @url-. Its purpose is to turn off autorouting at specific method or class level

What do you guys think?

[electrical]

I think it always can be useful to have the possibility to specify the route ( i use that sometimes )

[Luracast]

@electrical Yes! @url ... is already there for that purpose. now we will get another way to turn of auto routing (using @url-)

[electrical]

Ahh okay, thought that @url+ and @url- would replace @url :-) perhaps we need an other name then for disabling the auto routing?

[Luracast]

Is @url- a good enough name for auto route disabling or what else can be a better name?

[electrical]

perhaps this is useless but;

@routing auto = Auto routing enabled @routing manual = Manual routing rules ( thus auto routing disabled )

in the first case you can add extra routes with the @url statements in the second case only routes defined with the @url statement will be active ( thus not the routes that would be auto generated )

[roynasser]

Wouldnt it maybe be easier and quicker to have a configuration flag for the restler class itself to disable all automatic routing?

Because if you are writing new code, then you can just use __ in front of the function name, isnt that correct?

Maybe a $Restler->SetRouting(false); call could disable all default routes and manual ones can be added?

Because imho most people who write with restler from the bottom up either always use, or never use the auto routes... in my particular case I prefer to explicitly declare the routes with @url, so having a global disable would be great... For case-by-case I'd could have it auto-enabled and use __function() to declare the functions (as I do currently)...

[electrical]

@RVN-BR: At the moment i define all paths manually, so i understand your point of view. Your proposal looks like a good idea in my opinion.

[electrical]

http://blog.apigee.com/detail/restful_api_design_tips_for_handling_exceptional_behavior/

Error code handling and suppressing it also should be integrated.

[Luracast]

_suppress_responsecodes for Client supports limited HTTP methods

@electrical It is already in place :)

Auto routing ON/OFF

@RVN-BR I think auto routing on/off should be done at the class level to get the maximum flexibility and a doc comment is good enough to achieve this.

By default auto routing is turned on for all classes unless they add @auto-routing-off (new name for @url-. Suggest me with a better name) is added at the class level or method level.

I prefer not to have too many ways to achieve the same thing, as it may add to the confusion.

[electrical]

Very nice :-)

Perhaps too early, but do you have an idea how far you are with the new version ? ;-)

[Luracast]

@electrical half way through! :)

[electrical]

Cool :-) If you need any help, just met me know.

[rpip]

Great ideas. Just make sure you keep the library size as small as possible. That"s one of the things that has won many people over to Restler --- well, for me.

[electrical]

@mawuli-ypa I'm pretty sure that will stay the case. Most additions are very small or are optional.

[Luracast]

@mawuli-ypa as @electrical said it will always remain slim. Much of the muscle we are adding now will slightly increase processing load, only on the debug mode. Production mode will use a cache and will keep running efficiently as it used to be if not faster :)

Helping Restler Project

@electrical it will be great if you can go through all the questions asked about restler and come up with a FAQ page. You may use the wiki here for that purpose.

I will invite those who are interested to beta test Restler 3 once it is functional.

[electrical]

@Luracast No problem, give me a couple days to get some stuff online :-)

And of course i'm interested to beta test Restler 3 ;-)

[Luracast]

@electrical Great! looking forward for that :)

[electrical]

@Luracast Slowly filling it up: https://github.com/Luracast/Restler/wiki/FAQ Hope this is a bit in line what you had in mind.

[kzap]

i'd like to test out the documentation using swift too

for the FAQ, probably show also how to make a function require authentication and what the authentication class is, or just atleast put the question and link to the example.

also a good FAQ question is the how url routes with parameters work.

say

function test(param1,param2) {} = /test/param1/ or /test/param1/param2/
but @url /test/:param1/:param2/ != /test/param1/ , it has to be /test/param1/param2/ and you specify another route
@ @url /test/:param1/

also how to order functions like rewrite rules, you should put the more specific functions at the top of the file and the more general paths at the bottom

function testThis(){} should come before function test (){} and not after it

[electrical]

@kzap

Thanks for your idea's for the FAQ, i will implement those questions :-) ( in a few days )

[rpip]

Does the current Restler support file uploads and if so, how do i do it? If not, why not add it to the next version?

[roynasser]

Eagerly awaiting Restler 3.0 :)

Can you give us a brief overview of what, if anything, will need to be changed in existing scripts/classes?

I currently have a pretty advanced authentication scheme that ties in to our authentication DB, based on module/class and action that the user is trying to execute. It is cached on a Redis layer as well, etc, so as much as I can re-use of my code will make things easier to port to 3.0...

On the "classes" themselves, we have also started porting parts of our codebase. Is there something to watch out from the current version?

We utilize an object that we instantiate from our legacy system that brings with it a lot of info required to run our system. We currently have a Global call above the functions to bring that and our DB object into each necessary step. Can you comment on a better way of doing this? We cant rewrite all of the system due to practicality, so the we just imported these 2 objects, but I wonder if this is the best/most memory efficient way of doing it? And also, just because it works now, doesnt mean it will work on restler 3, which worries me as well... If you can comment on suggested changes or whether this should be ok (despite maybe not being 100% oop by the book, whatever - this worries me much less, we run a real production system, so some things need to have a line drawn and we can settle for "works great but isnt as purdy as one would like"...)

[electrical]

@RVN-BR I doubt that the core functionality will change much. It's mainly adding some nice stuff to it.

[electrical]

@mawuli-ypa That's a very good one. That shouldn't be to difficult to achieve. A single file upload would be pretty easy. How to handle multiple files? or is that not needed?

[rpip]

@electrical Yes, multiple file uploads. Usually, multiple file uploads are not well ordered in the $_FILES array. We can use a function like this to reorder the $_FILES array then pass it to the $data_array:

 function arrange_files($file_post) {
           //$file_post is $_FILES;
             $files_array = array();
             $file_count = count($file_post['name']);
             $file_keys = array_keys($file_post);

         for ($i=0; $i<$file_count; $i++) {
              foreach ($file_keys as $key) {
              $files_array[$i][$key] = $file_post[$key][$i];
          }
      }
      return $files_array;
  }

[electrical]

I think for fileuploads we should take a careful look at it to implement it correctly. things like validation are very important ( size, extension, etc ) @Luracast Do you already have an idea about this?

@mawuli-ypa: thanks for the code, i'm sure it's gonna help us implementing it; if you have any additional idea's about it or other things please share :D

[roynasser]

Hi Luracast,

any news? I'm anxious :)