How can I avoid request parameter names getting encoded when making a form-urlencoded POST request?

[RacZo]

Hello all!

First of all, RetroFit and RetroFit 2.0 are awesome, I've used them in several Android Apps in the past couple of years with great success, well, those Apps had decent backend APIs... but now... I'm working with this horribly designed API (made with ruby on rails) and I can't change the API. This nightmarish API has (among other awful things) POST methods with parameters such as:

• user[email]
• location[address]
• location[city]

And I've created this method to consume one of the methods:

@FormUrlEncoded
@POST("userExists.json")
Call<ExistsResponse> emailExists(@Field("user[email]") String email);

My service is created with a GsonConverterFactory with the following GSON object:

            Gson gson = new GsonBuilder()
                    .excludeFieldsWithModifiers(Modifier.TRANSIENT)
                    .setDateFormat(WebService.API_DATE_FORMAT)
                    .disableHtmlEscaping()
                    .create();

I read that using disableHtmlEscaping() on the gson object would help but it didn't.

The problem I'm having is that the square brackets in the parameter name is getting encoded like this:

... D/OkHttp: user%5Bemail%5D=email%40example.com

(Yes!, I'm using the very neat HttpLoggingInterceptor to log the body of the requests!)

This is driving me crazy, I've tried all the possible ways to make this request and the backend API keeps sending me 404 because it is not understanding the request parameter names.

Is there a way to tell retrofit not to encode the parameter names in the body of a post request?

Your help will be greatly appreciated!

Happy holidays!

[artem-zinnatullin]

AFAIK: Passed string value is not going through converter (Gson) since it's a Form parameter. Try @Field(encoded = true)

On Thu, Dec 24, 2015, 05:03 Oscar S. notifications@github.com wrote:

Hello all!

First of all, RetroFit and RetroFit 2.0 are awesome, I've used them in several Android Apps in the past couple of years with great success, well, those Apps had decent backend APIs... but now... I'm working with this horribly designed API (made with ruby on rails) and I can't change the API. This nightmarish API has (among other awful things) POST methods with parameters such as:

• user[email]
• location[address]
• location[city]

And I've created this method to consume one of the methods:

@FormUrlEncoded @POST("userExists.json")Call emailExists(@Field("user[email]") String email);

My service is created with a GsonConverterFactory with the following GSON object:

        Gson gson = new GsonBuilder()
                .excludeFieldsWithModifiers(Modifier.TRANSIENT)
                .setDateFormat(WebService.API_DATE_FORMAT)
                .disableHtmlEscaping()
                .create();

I read that using disableHtmlEscaping() on the gson object would help but it didn't.

The problem I'm having is that the square brackets in the parameter name is getting encoded like this:

... D/OkHttp: user%5Bemail%5D=email%40example.com

(Yes!, I'm using the very neat HttpLoggingInterceptor https://github.com/square/okhttp/tree/master/okhttp-logging-interceptor to log the body of the requests!)

This is driving me crazy, I've tried all the possible ways to make this request and the backend API keeps sending me 404 because it is not understanding the request parameter names.

Is there a way to tell retrofit not to encode the parameter names in the body of a post request?

Your help will be greatly appreciated!

Happy holidays!

— Reply to this email directly or view it on GitHub https://github.com/square/retrofit/issues/1407.

@artem_zin

[RacZo]

Digging around, I found this commit 4c38147 made by @JakeWharton that adds methods to tell if the field name or value should be encoded or not in the body.

But, I can't find a way to use them... will keep trying.

UPDATE: Ok... apparently this worked at some point in retrofit one, not retrofit2. :(

[JakeWharton]

There is a test case that proves encoded=true works: https://github.com/square/retrofit/blob/90729eb2ae2f3329281c1f9813ab1de3daa71ad0/retrofit/src/test/java/retrofit2/RequestBuilderTest.java#L1488-L1498. What output do you get when you use it?

[RacZo]

Hey thank you @JakeWharton, you rock man!,

Evidently I missed that test case. I tried with:

@Field(value = "user[email]", encoded = false)

and the request didn't went through, then I tried with:

@Field(value = "user[email]", encoded = true)

and that was all I had to do in order to get a response from this API.

On the other hand, there seems to be an issue with the HttpLoggingInterceptor because the output in my log is still:

... D/OkHttp: user%5Bemail%5D=email%40example.com

[hatcher521]

i have meet this question,too.forexample:Date type，brackets

[JakeWharton]

Stop spamming our issues. This is your one and only warning.

(Their comment was since deleted)

On Fri, Jun 3, 2016 at 12:41 AM AMIT SHEKHAR notifications@github.com wrote:

You can use this library . This library supports this Android Networking https://github.com/amitshekhariitbhu/AndroidNetworking

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/square/retrofit/issues/1407#issuecomment-223489318, or mute the thread https://github.com/notifications/unsubscribe/AAEEEX_rEgq_ZiW4cDdz0WLykbVoc4pgks5qH7CDgaJpZM4G64vQ .

[sriramji]

I am facing the same issue after upgrading to Retrofit 2.0

Tried Both : @Field(value = "email", encoded = false) String email @Field(value = "email", encoded = true) String email

Getting same result "sriramji.k%40gmail.com", but i want "sriramji.k@gmail.com"

[sriramji]

@JakeWharton Please help me out of this issue

[sriramji]

This issue can be fix if you add one more method in Formbody.Builder like

public static final class Builder {
......
public Builder addPlainText(String name, String value) {
      names.add(name);
      values.add(value);
      return this;
    }
.....
}

[mzander]

I ran into the same issue since I upgraded from retrofit 1.9 to retofit 2.

@Field(value = "videos[]") List<Integer> videoIds Result: videos%5B%5D=4340934

@Field(value = "videos[]", encoded = true) List<Integer> videoIds Result: videos%5B%5D=4340934

I am not sure how the encoded parameter works but I guess this is only for the values of the field?

@sriramji did you find any solution for this?

[sriramji]

@mzander For now i created my own FormBody.Builder like i said before

 @POST(LOGIN_URL)
 Call<BaseResponse> loginUser(@Body RequestBody body);

 RequestBody formBody = new FormBody.Builder()
                .add(EMAIL, emailId.getText().toString())
                .add(PASSWORD, password.getText().toString())
                .build();

 Call<BaseResponse> loginCall = RetorfitService.service.loginUser(formBody);

[ylfzq]

I've read the source code of Retrofit and Okhttp, here is the key:

// in package okhttp3
public final class HttpUrl {
  static final String QUERY_ENCODE_SET = " \"'<>#";
  static final String QUERY_COMPONENT_ENCODE_SET = " \"'<>#&=";
  static final String QUERY_COMPONENT_ENCODE_SET_URI = "\\^`{|}";
  static final String FORM_ENCODE_SET = " \"':;<=>@[]^`{}|/\\?#&!$(),~";

  static void canonicalize(Buffer out, String input, int pos, int limit, String encodeSet,
      boolean alreadyEncoded, boolean strict, boolean plusIsSpace, boolean asciiOnly) {
    Buffer utf8Buffer = null; // Lazily allocated.
    int codePoint;
    for (int i = pos; i < limit; i += Character.charCount(codePoint)) {
      codePoint = input.codePointAt(i);
      if (alreadyEncoded
          && (codePoint == '\t' || codePoint == '\n' || codePoint == '\f' || codePoint == '\r')) {
        // Skip this character.
      } else if (codePoint == '+' && plusIsSpace) {
        // Encode '+' as '%2B' since we permit ' ' to be encoded as either '+' or '%20'.
        out.writeUtf8(alreadyEncoded ? "+" : "%2B");
      } else if (codePoint < 0x20
          || codePoint == 0x7f
          || codePoint >= 0x80 && asciiOnly
          || encodeSet.indexOf(codePoint) != -1
          || codePoint == '%' && (!alreadyEncoded || strict && !percentEncoded(input, i, limit))) {
        // Percent encode this character.
        if (utf8Buffer == null) {
          utf8Buffer = new Buffer();
        }
        utf8Buffer.writeUtf8CodePoint(codePoint);
        while (!utf8Buffer.exhausted()) {
          int b = utf8Buffer.readByte() & 0xff;
          out.writeByte('%');
          out.writeByte(HEX_DIGITS[(b >> 4) & 0xf]);
          out.writeByte(HEX_DIGITS[b & 0xf]);
        }
      } else {
        // This character doesn't need encoding. Just copy it over.
        out.writeUtf8CodePoint(codePoint);
      }
    }
}

Draw a conclusion is that, @Query won't encode char '[' and ']' into percent-encoding, but @Field will. So if you use @Query("user[email]"), it will be ok, but @Field("user[email]") will fail

[adriamt]

Any clue how to solve this problem? I'm facing the same issue. I have an API call like this:

@FormUrlEncoded
@POST(Constantes.URL_AUTHENTICATE)
Call<Object> authenticateUser(@Field("name") String name , @Field("password") String pwd);

And when the password parameter is an encoded string, for example, MTIzNA== and when I make the request this String becomes MTIzNA%3D%3D.

I've tried what @sriramji says and I've used a FormBody builder like this

 RequestBody formBody = new FormBody.Builder()
  .add("name", etUserName.getText().toString())
  .add("password", new String(encodeValue))
.build();

But didn't work either. I've searched a lot but I didn't find anything. Any help will be appreciated.

[Nan0fm]

Anybody solve this problem? please help...))

[adriamt]

I've done this to solve the problem. I have an Interceptor on my httpclient and when the method equals to post or put I decode de body. This is my code.

       HttpLoggingInterceptor logging = new HttpLoggingInterceptor();
        logging.setLevel(HttpLoggingInterceptor.Level.BODY);
        OkHttpClient.Builder httpClient = new OkHttpClient.Builder();
        httpClient.addInterceptor(new Interceptor() {
            @Override
            public Response intercept(Interceptor.Chain chain) throws IOException {
                Request original = chain.request();

                //El problema es que codifica los parametros del body y no los queremos codificados
                String postBody = bodyToString(original.body());
                String newPostBody = URLDecoder.decode(postBody);
                RequestBody body = original.body();
                RequestBody requestBody = null;

                if(body!=null){
                    requestBody = RequestBody.create(original.body().contentType(),newPostBody);
                }

                // Aqui el problema es que se tiene que modificar el body en los put y los post para decodificar,
                // pero los get y delete no tienen body
                Request request;
                if (original.method().equals("post")) {
                    request = original.newBuilder()
                            .method(original.method(), original.body())
                            .post(requestBody)
                            .build();
                }else if(original.method().equals("put")){
                    request = original.newBuilder()
                            .method(original.method(), original.body())
                            .put(requestBody)
                            .build();
                }else{
                    request = original.newBuilder()
                            .method(original.method(), original.body())
                            .build();
                }

                return chain.proceed(request);
            }

            public String bodyToString(final RequestBody request){
                try {
                    final RequestBody copy = request;
                    final Buffer buffer = new Buffer();
                    if(copy != null)
                        copy.writeTo(buffer);
                    else
                        return "";
                    return buffer.readUtf8();
                }
                catch (final IOException e) {
                    return "did not work";
                }
            }
        });

        httpClient.addInterceptor(logging);

[Tindi]

@adriamt what does your API Interface look like? Tried your solution but no success.

[adriamt]

I've got two classes, my interface :

public interface MyApi{
    @FormUrlEncoded
    @POST(Constantes.URL_USER)
    Call<Object> createUser(
            @Field(value="name") String name,
            @Field(value="password") String pwd);
}

I have other calls, but all look like this and then my RestClient:

public class MyRestClient {
    public static MyApi REST_CLIENT;
    private static String ROOT = BuildConfig.API_HOST;

    static {
        setupRestClient();
    }

    public MyRestClient() {}

    private static void setupRestClient() {

        HttpLoggingInterceptor logging = new HttpLoggingInterceptor();
        logging.setLevel(HttpLoggingInterceptor.Level.BODY);
        OkHttpClient.Builder httpClient = new OkHttpClient.Builder();
        httpClient.addInterceptor(new Interceptor() {
            @Override
            public Response intercept(Interceptor.Chain chain) throws IOException {
                Request original = chain.request();

                String postBody = bodyToString(original.body());
                String newPostBody = URLDecoder.decode(postBody);
                RequestBody body = original.body();
                RequestBody requestBody = null;

                if(body!=null){
                    requestBody = RequestBody.create(original.body().contentType(),newPostBody);
                }

                Request request;
                if (original.method().equals("post")) {
                    request = original.newBuilder()
                            .method(original.method(), original.body())
                            .post(requestBody)
                            .build();
                }else if(original.method().equals("put")){
                    request = original.newBuilder()
                            .method(original.method(), original.body())
                            .put(requestBody)
                            .build();
                }else{
                    request = original.newBuilder()
                            .method(original.method(), original.body())
                            .build();
                }

                return chain.proceed(request);
            }

            public String bodyToString(final RequestBody request){
                try {
                    final RequestBody copy = request;
                    final Buffer buffer = new Buffer();
                    if(copy != null)
                        copy.writeTo(buffer);
                    else
                        return "";
                    return buffer.readUtf8();
                }
                catch (final IOException e) {
                    return "did not work";
                }
            }
        });

        httpClient.addInterceptor(logging);

        OkHttpClient client = httpClient
                .readTimeout(60, TimeUnit.SECONDS)
                .writeTimeout(60, TimeUnit.SECONDS)
                .build();

        Gson gson = new GsonBuilder().registerTypeAdapter(Date.class, new DateDeserializer()).setLenient().create();

        Retrofit retrofit = new Retrofit.Builder()
                .baseUrl(ROOT)
                .addConverterFactory(GsonConverterFactory.create(gson))
                .client(client)
                .build();

        REST_CLIENT = retrofit.create(MyApi.class);
    }

    public static MyApi get() {
        return REST_CLIENT;
    }

    public staticMyApi post() {
        return REST_CLIENT;
    }

    public static MyApi put() {
        return REST_CLIENT;
    }

    public static MyApi delete() {
        return REST_CLIENT;
    }
}

I don't know if its the best solution but it works for me for the moment.

[NeLk42]

A very simple solution is to pass the value encoded and then add the encoded = true flag to retrofit.

String item = "MTUwNTIyODgxMDg4Mw==";

encodedItem = URLEncoder.encode(item, "utf-8");

@Query(value = "item", encoded = true) String item

This way when the time comes it'll decode it.

[Nan0fm]

@NeLk42 but how I can use your solution in POST parameters? I need send params like this ( key : value) login[name] : some-name date : 2017-08-28T12:12:12+0200 and request looks like

@FormUrlEncoded
@POST(urlLogin)
Call<Login> signIn(@Field("login[name]") String name,      
                               @Field("date")   String date);

@adriamt I'm had trying your solution, but when I changing request body, then I can't parse back them to key-value params and I don't know what I 'm sending ....

I don't know what i'm doing wrong .. :(

[NeLk42]

@Nan0fm Can you try this?

Before passing the value, encode it.

String username = getUsername();
String encodedUsername = URLEncoder.encode(username, "utf-8");
retrofitObject.signIn(encodedUsername, date)

Let retrofit know that you've encoded that value.

@FormUrlEncoded
@POST(urlLogin)
Call<Login> signIn(@Field("login[name]", encoded = true) String name,      
                               @Field("date")   String date);

In my case, I'm using it to validate a SHA512 encoded password against a signed header, I don't see why it shouldn't work for you.

[bbfxier]

@sriramji can you give a full code? because you add a method called addPlainText then your demo use add method. is it your new added method?

[sriramji]

yeah that was my added method. It is not recommended but I have no other choice, so i created my own custom FormBody class

This is the class that we need to look

package okhttp3;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import okhttp3.internal.Util;
import okio.Buffer;
import okio.BufferedSink;

public final class FormBody extends RequestBody {
    private static final MediaType CONTENT_TYPE = MediaType.parse("application/x-www-form-urlencoded");
    private final List<String> encodedNames;
    private final List<String> encodedValues;

    private FormBody(List<String> encodedNames, List<String> encodedValues) {
        this.encodedNames = Util.immutableList(encodedNames);
        this.encodedValues = Util.immutableList(encodedValues);
    }

    public int size() {
        return this.encodedNames.size();
    }

    public String encodedName(int index) {
        return (String)this.encodedNames.get(index);
    }

    public String name(int index) {
        return HttpUrl.percentDecode(this.encodedName(index), true);
    }

    public String encodedValue(int index) {
        return (String)this.encodedValues.get(index);
    }

    public String value(int index) {
        return HttpUrl.percentDecode(this.encodedValue(index), true);
    }

    public MediaType contentType() {
        return CONTENT_TYPE;
    }

    public long contentLength() {
        return this.writeOrCountBytes((BufferedSink)null, true);
    }

    public void writeTo(BufferedSink sink) throws IOException {
        this.writeOrCountBytes(sink, false);
    }

    private long writeOrCountBytes(BufferedSink sink, boolean countBytes) {
        long byteCount = 0L;
        Buffer buffer;
        if(countBytes) {
            buffer = new Buffer();
        } else {
            buffer = sink.buffer();
        }

        int i = 0;

        for(int size = this.encodedNames.size(); i < size; ++i) {
            if(i > 0) {
                buffer.writeByte(38);
            }

            buffer.writeUtf8((String)this.encodedNames.get(i));
            buffer.writeByte(61);
            buffer.writeUtf8((String)this.encodedValues.get(i));
        }

        if(countBytes) {
            byteCount = buffer.size();
            buffer.clear();
        }

        return byteCount;
    }

    public static final class Builder {
        private final List<String> names = new ArrayList();
        private final List<String> values = new ArrayList();

        public Builder() {
        }

        public FormBody.Builder add(String name, String value) {
            this.names.add(HttpUrl.canonicalize(name, " \"':;<=>@[]^`{}|/\\?#&!$(),~", false, false, true, true));
            this.values.add(HttpUrl.canonicalize(value, " \"':;<=>@[]^`{}|/\\?#&!$(),~", false, false, true, true));
            return this;
        }

        public FormBody.Builder addEncoded(String name, String value) {
            this.names.add(HttpUrl.canonicalize(name, " \"':;<=>@[]^`{}|/\\?#&!$(),~", true, false, true, true));
            this.values.add(HttpUrl.canonicalize(value, " \"':;<=>@[]^`{}|/\\?#&!$(),~", true, false, true, true));
            return this;
        }

        public FormBody build() {
            return new FormBody(this.names, this.values);
        }
    }
}

In Builder Inner class add your custom method

public FormBody.Builder addPlainText(String name, String value) {
            this.names.add(name);
            this.values.add(value);
            return this;
}

[hubangmao]

The problem was finally solved ！

add @Headers("Content-Type:application/x-www-form-urlencoded; charset=utf-8")

[KomoriWu]

还是不行

[jial-apa]

Neither the extra content-type header nor setting the encoded to false works. The fields still got encoded.

[michaelwiles]

The Field(encoded = true) directive is ONLY there for the scenario that the value you're sending is already encoded. Setting encoded = true does NOT disable encoding it simply ensures the value is not double encoded.

Unfortunately though, as has been mentioned, a lot of people want to turn the encoding off altogether. focussing on the behaviour of the encoded option in the Field is not the place to get this as this is not by design, for turning the encoding off - it is only to stop double encoding.

A feature request should rather be logged to add a flag to turn off encoding altogether via a @Field annotation.

[pmashelkar]

I am passing a complex POJO @POST("project/{pro_id}/sender") Single uploadRenderingJSON( @Path("pro_id") String proId, @Body RenderRequest renderRequest);

Some fields of the POJO are already encoded using StringEscapeUtils.escapeJava(textCaption);

How can I avoid the strings getting encoded again. Since double quotes and backslash are converted to \\" and \\ respectively.

Please suggest.

[yusufonderd]

I'm facing same situation.

[BoukhariAyoub]

I'm having this problem and nothing of the solutions above is working

[Sainathhiwale]

[ { "ofsNo": "180007", "dispatchDate": "07/04/2018", "vehicleNo": "ka45p3654", "transporterName": "trns", "depotName": "KSBCL", "ofsId": 1 }, { "ofsNo": "180004", "dispatchDate": "07/04/2018", "vehicleNo": "KA09B6435", "transporterName": "trns", "depotName": "KSBCL", "ofsId": 10006 } ]

Hi every one i want get ofsno in spinner and base on ofsno populate value into EditText like as "dispatchDate": "07/04/2018", "vehicleNo": "KA09B6435", "transporterName": "trns", "depotName": "KSBCL", etc but using retrofit

public void networkCall(){ ofsworld = new ArrayList(); // Create an array to populate the spinner ofsIdArrayList = new ArrayList(); final ApiInterface apiInterface = ApiClient.getClient().create(ApiInterface.class); Call<List> call =apiInterface.getOfsID(); call.enqueue(new Callback<List>() { @Override public void onResponse(Call<List> call, Response<List> response) { if (response.isSuccessful()){ Log.d("data",String.valueOf(response)); }

           try {
                List<OfsId> ofsIds = response.body();
                for (int i=0;i<ofsIds.size();i++){
                    if (i==0){
                        //String code = ofsIds.get(i).getOfsNo();
                        String leaveType = ofsIds.get(i).getOfsNo();
                        String dipatchDate = ofsIds.get(i).getDispatchDate();
                        String depotName = ofsIds.get(i).getDepotName();
                        String vechicleNo = ofsIds.get(i).getVehicleNo();
                        String trnsName = ofsIds.get(i).getTransporterName();
                        ofsIdArrayList.add(leaveType);
                        et_DispatchDate.setText(dipatchDate);
                        et_DepotName.setText(depotName);
                        et_VechicleNo.setText(vechicleNo);
                        et_TransporterName.setText(trnsName);

                    }

                    arrayAdapter = new ArrayAdapter(MainActivity.this,android.R.layout.simple_spinner_item,ofsIdArrayList);
                    arrayAdapter.setDropDownViewResource(android.R.layout.simple_spinner_dropdown_item);
                    ofs_Spinner.setAdapter(arrayAdapter);
                }

            }catch (Exception e){
                e.printStackTrace();
            }
        }

        @Override
        public void onFailure(Call<List<OfsId>> call, Throwable t) {

        }
    });
}

[394090466]

There is still no solution

[sixangle]

I have tried most of the solutions above, but no one works. It took me almost half of the day to finally resolve this problem.

Api interface : (Note: do not use the @FormUrlEncoded annotation.)

@POST("/jdai/snapshop")
@Headers("Content-Type:text/plain")
Call<JDAiResultBean> postServer(@QueryMap Map<String, String> queryMap, @Body RequestBody body); 

Build your post parameters like as below.

String url = "channel_id=" + CHANNEL_ID  + "&imgBase64=" + mImageBase64 + "&topK=1";
MediaType mediaType = MediaType.parse("text/plain");
RequestBody body = RequestBody.create(mediaType, url);
api.postServer(queryMap, body);

That's all. Hope this could save your time.

[FrancisRR]

Please Use encoded = true @Query(value = "email", encoded = true)

@HTTP(method = "DELETE", path = "/endurl", hasBody = true) Observable deleteContact(@Query(value = "email", encoded = true) String email);

[drod3763]

I am running into this now, and none of the solutions are working. It's encoding an email and password in a form and now it's throwing errors.

[SmartAppsDevelopment]

i have tried all above solution no solution worked for my problem i am sending encrypted params in POST request witch add special character during encoding and OKHTTP adding special codes as mention above