Implement simplified <form> to Object handling using JSON for POST requests

[murphye]

Description Developers are moving to a REST-centric world, along with SPA (i.e. Angular). It is very common to use JavaScript to generate JSON from a <form> and submit it via AJAX. However, this is very complicated and error prone.

With the addition of Qute, Quarkus would benefit from simplified <form> submission mechanism that is reliant on JSON rather than @Form in JAX-RS or @ModelAttribute in Spring (which is not even supported in Quarkus anyways). The mechanism would be completely server-side and not dependent on JavaScript.

Benefits are:

• Common deserialization mechanism of both <form>s and JSON data for POJOs
• Lower learning curve for developers especially for Qute
• Simplified form handling mechanism for developers who struggle with JAX-RS and Spring APIs (pretty much everyone)
• Alternative to implementing @ModelAttribute in Spring
• Covers the vast majority of <form> use cases

Potential drawbacks are:

• May not cover corner cases for complex form data. I can't think of any off the top of my head
• Does not offer companion form validation API for advanced form error handling (see Spring BindingResult)
• Proposed implementation is not very obvious to developer as there is no "magic annotation" required

Implementation ideas The functionality I am describing does not require an API change. It just needs a ContainerRequestFilter or similar mechanism to be included with Quarkus. I have created a proof of concept here: FormDataToJsonFilter.java

It works by:

1. Checks if resource method consumes both APPLICATION_FORM_URLENCODED_TYPE and APPLICATION_JSON_TYPE. This is the key for enabling this feature.
2. Checks if it's a POST request and has a content type of APPLICATION_FORM_URLENCODED_TYPE
3. Converts the getEntityStream to a String. This contains the body of the POST request which are URL parameters in query string format.
4. Uses Netty's QueryStringDecoder to convert to a Map.
5. Uses Jackson's ObjectMapper to convert the Map to JSON string.
6. Changes body of request to be JSON using setEntityStream
7. Change "Content-Type" header to be MediaType.APPLICATION_JSON

Sample implementation:

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.ObjectMapper;
import io.netty.handler.codec.http.QueryStringDecoder;
import io.vertx.core.http.HttpServerRequest;
import org.jboss.logging.Logger;
import org.jboss.resteasy.core.ResourceMethodInvoker;

import javax.inject.Inject;
import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.container.ContainerRequestFilter;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriInfo;
import javax.ws.rs.ext.Provider;
import java.io.ByteArrayInputStream;
import java.util.List;
import java.util.Map;

@Provider
public class FormDataToJsonFilter implements ContainerRequestFilter {

    private static final Logger LOG = Logger.getLogger(FormDataToJsonFilter.class);

    @Context
    UriInfo info;

    @Context
    HttpServerRequest request;

    @Inject
    ObjectMapper objectMapper;

    @Override
    public void filter(ContainerRequestContext context) {
        // Get a handle on the Method to be invoked from this HTTP request
        ResourceMethodInvoker resourceMethodInvoker = (ResourceMethodInvoker)context.getProperty("org.jboss.resteasy.core.ResourceMethodInvoker");

        // Filter based on the method consuming both Form and JSON
        boolean hasForm = false;
        boolean hasJson = false;

        // Cycle through consumes MediaTypes
        for(MediaType mediaType : resourceMethodInvoker.getConsumes()) {
            if(mediaType.equals(MediaType.APPLICATION_FORM_URLENCODED_TYPE)) {
                hasForm = true;
            }
            else if(mediaType.equals(MediaType.APPLICATION_JSON_TYPE)) {
                hasJson = true;
            }
        }

        // For all form POST requests that have consumes both Form and JSON and content of APPLICATION_FORM_URLENCODED_TYPE
        if(context.getMethod().equals("POST")
                && hasForm && hasJson
                && context.getMediaType().equals(MediaType.APPLICATION_FORM_URLENCODED_TYPE)) {

            // Pull out the POST query string (the entity) from the request body
           String entity = convertStreamToString(context.getEntityStream());

           // Decode the query string and convert to a Map
           Map<String, List<String>> params = new QueryStringDecoder(entity, false).parameters();

           // Make sure arrays are unwrapped appropriately with global objectMapper when JSON is converted to POJO
           objectMapper.enable(DeserializationFeature.UNWRAP_SINGLE_VALUE_ARRAYS);

           try { //  Write out the JSON string from the Map
                String jsonResult = objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(params);

                // Write the bytes for the JSON back into the request
                context.setEntityStream(new ByteArrayInputStream(jsonResult.getBytes()));

                // Change the Content-Type so the request can be correctly consumed
                context.getHeaders().putSingle("Content-Type", MediaType.APPLICATION_JSON);

                // Now the HTTP POST request will be processed as JSON data rather than form data.
            } catch (JsonProcessingException e) {
                LOG.error("Error converting POST body to JSON object.", e);
                context.abortWith(Response.status(Response.Status.BAD_REQUEST).build());
            }
        }
    }

    private static String convertStreamToString(java.io.InputStream is) {
        java.util.Scanner s = new java.util.Scanner(is).useDelimiter("\\A");
        return s.hasNext() ? s.next() : "";
    }
}

Here is how to use this functionality in Spring Web (notice there are 2 consumes):

@PostMapping(path = "/owners/{ownerId}/edit", consumes = {MediaType.APPLICATION_FORM_URLENCODED_VALUE, MediaType.APPLICATION_JSON_VALUE })
public ResponseEntity processUpdateOwnerForm(Owner owner, @PathVariable int ownerId) {...}

Here is how to use it in JAX-RS (notice there are 2 consumes):

@POST
@Path("/owners/{ownerId}/edit")
@Consumes({MediaType.APPLICATION_FORM_URLENCODED, MediaType.APPLICATION_JSON})
public TemplateInstance processUpdateOwnerForm(Owner owner, @PathParam("ownerId") int ownerId) {...}

Existing Solutions Just for reference, here are the roughly equivalent usages with Spring and JAX-RS (there is usually more involved than this):

Spring using @ModelAttribute (doesn't work in Quarkus):

@PostMapping(path = "/owners/{ownerId}/edit", consumes = {MediaType.APPLICATION_FORM_URLENCODED_VALUE })
public ResponseEntity processUpdateOwnerForm(@ModelAttribute Owner owner, @PathVariable int ownerId) {...}

JAX-RS using @Form (also requires using @FormItem on the model object):

@POST
@Path("/owners/{ownerId}/edit")
@Consumes({MediaType.APPLICATION_FORM_URLENCODED})
public TemplateInstance processUpdateOwnerForm(@Form Owner owner, @PathParam("ownerId") int ownerId) {...}

Conclusion

I feel this should be included in Quarkus (rather than Resteasy) because it's an opinionated approach with a relatively simple solution as a standard filter (could be disabled through config too). This could be offered as the default mechanism for processing form data with Qute and Quarkus.

I hope you will consider adding it in the near future as it would ease some current pains with adopting the Spring APIs for form processing with Qute.

Further Notes

• Verified this works with <form><select multiple> items to convert to a List<String>.

[nimo23]

Sounds good. Plz consider security constraints for Qute-Form (for example, JavaServerFaces has built in support for CSRF, etc.).

Does not offer companion form validation API

As we can already bind hibernate validation api to model or method, we can respond with validation errors by http response to the client.

Uses Jackson's ObjectMapper to convert the Map to JSON string.

Json-B?

[murphye]

@nimo23 What you speak of is interesting (security, validation), but is out of scope for this issue. This issue is only for the Form -> Object handling.

Side note: I have verified that @Valid works, but isn't very useful for form handling. I have ended up using the Validator API programmatically in my demo (work in progress) as shown here: https://github.com/murphye/spring-petclinic/blob/spring-boot-1.5-to-quarkus/src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java#L158

What is offered by Spring is an abstraction on how the validation errors are captured with BindingResult. I have no interest in Quarkus replicating this functionality.

[murphye]

@nimo23 I am not sure about JSON-B. I will say that Jackson is very intelligent on how it serializes and deserializes that make the sample implementation very straightforward. I do not know if JSON-B will work the same way.

[murphye]

@mkouba I would be interested to get your take on this proposal because it primarily targets users of Qute.

[nicmarti]

I really +1 this idea. I tried to implement a similar solution here

I reach a point where I think that we could really look at what Play2 did with the Java Form and a FormFactory. See the documentation, with Bean Validation and JSR-380. The idea is to define a Form<Owner> and not to unmarshall the entity Owner in your controller. This is then easier to return badRequest, or to re-display a template with errors (as we both did actually).

Let's keep posted and see if this could generate some ideas for Qute.

[mageddo]

Description Developers are moving to a REST-centric world, along with SPA (i.e. Angular). It is very common to use JavaScript to generate ......

Very great solution, I've inspired on your code and made a fork which only is triggered when the request method is annotated with FormDataJson.java, hope it helps.

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * Methods using that annotation will have form data parsed to VO classes using Jackson Json
 * deserializer
 */
@Target({ElementType.PARAMETER, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
public @interface FormDataJson {
}

import java.io.IOException;
import java.io.InputStream;
import java.lang.annotation.Annotation;
import java.lang.reflect.Type;

import javax.ws.rs.Consumes;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.MultivaluedMap;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriInfo;
import javax.ws.rs.ext.MessageBodyReader;
import javax.ws.rs.ext.Provider;

import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.ObjectMapper;

import org.jboss.resteasy.spi.util.FindAnnotation;

import io.netty.handler.codec.http.QueryStringDecoder;
import io.vertx.core.http.HttpServerRequest;
import lombok.extern.slf4j.Slf4j;

@Slf4j
@Provider
@Consumes({MediaType.APPLICATION_FORM_URLENCODED})
public class FormDataToJsonAdapterBodyReader implements MessageBodyReader<Object> {

  private final ObjectMapper objectMapper = new ObjectMapper()
      .enable(DeserializationFeature.UNWRAP_SINGLE_VALUE_ARRAYS);

  @Override
  public boolean isReadable(Class<?> type, Type genericType,
      Annotation[] annotations, MediaType mediaType) {
    return FindAnnotation.findAnnotation(annotations, FormDataJson.class) != null
        || type.isAnnotationPresent(FormDataJson.class);
  }

  @Override
  public Object readFrom(Class<Object> type, Type genericType, Annotation[] annotations,
      MediaType mediaType, MultivaluedMap<String, String> httpHeaders, InputStream entityStream)
      throws WebApplicationException {
    final var entity = convertStreamToString(entityStream);
    final var params = new QueryStringDecoder(entity, false).parameters();
    try {
      final var multiValueJsonBytes = this.objectMapper.writeValueAsBytes(params);
      return this.objectMapper.readValue(multiValueJsonBytes, type);
    } catch (IOException e) {
      log.warn("status=Error converting POST form body to JSON object.", e);
      throw new WebApplicationException(Response
          .status(Response.Status.BAD_REQUEST)
          .entity(e.getMessage())
          .build());
    }
  }

  private static String convertStreamToString(java.io.InputStream is) {
    java.util.Scanner s = new java.util.Scanner(is).useDelimiter("\\A");
    return s.hasNext() ? s.next() : "";
  }
}

[geoand]

Very interesting indeed.

@FroMage what's your take on this? Should we add something like this to RESTEasy Reactive?

I'd also like to hear @edeandrea's take on it.

[edeandrea]

Personally the idea of being able to bind a form with fields similar to a json structure is beneficial. I've always liked that Spring had that ability and that I did not have to create some brand new object just because I wanted to form post similar/same data.

That being said there are some things to keep in mind that I think are needed in order to make it a proper solution.

1. Someone mentioned csrf above. To me that is required - otherwise people could get themselves into trouble.

2. What about multi-form posting? In the Spring example I could use @ModelAttribute across multiple POSTs and collect data across a set of forms, then "commit" the form on the last POST. Spring uses @SessionAttributes for this, which would open up a whole other can of worms - how to persist the state across multiple http requests. I think Spring supports both session-based and flash-based. I remember seeing something re: flash attributes in Quarkus in @FroMage 's Renard presentation a few weeks ago.

3. Adding onto the multi-view form posting - you would definitely need a way to handle input validation at each "step", vs an all at once. You might be able to do that with validation groups, but I'm not as familiar with doing it in jax-rs. All my experience is with Spring (and I can certainly share how we did it of needed).

Speaking of Renard - seems to me this all might be a good fit for that?