PUT,POST,PATCH with original values gives an error

[Dragomir-Ivanov]

Working with PATCH, I hit an issue doing updates onto resources: This is what I do:

GET http://localhost:8080/api/visits/5a4d69264e2127a7118b4c51

Result:
{
  "appointmentDT": "2018-01-05T18:00:00+02:00",
  "clientId": "5a4d63fe41095f105dcd49b2",
  "createdDT": "2018-01-04T01:37:18.079+02:00",
  "duration": 180,
  "id": "5a4d69264e2127a7118b4c51",
  "name": "Dryer, Refrigerator, Microwave Diagnose",
  "serviceId": "5a4d63fe41095f105dcd49b6",
  "technitian": "Asparuh Vasilev",
  "zipCode": "89107"
}

Then I try to update with the same exact data:

PUT http://localhost:8080/api/visits/5a4d69264e2127a7118b4c51

{
  "appointmentDT": "2018-01-05T18:00:00+02:00",
  "clientId": "5a4d63fe41095f105dcd49b2",
  "createdDT": "2018-01-04T01:37:18.079+02:00",
  "duration": 180,
  "id": "5a4d69264e2127a7118b4c51",
  "name": "Dryer, Refrigerator, Microwave Diagnose",
  "serviceId": "5a4d63fe41095f105dcd49b6",
  "technitian": "Asparuh Vasilev",
  "zipCode": "89107"
}

Result:
{
  "code": 422,
  "issues": {
    "createdDT": [
      "read-only"
    ],
    "id": [
      "read-only"
    ]
  },
  "message": "Document contains error(s)"
}

This shouldn't be the case since, I am putting exactly the same data, and removing read-only fields from the WEB UI, seems tedious and overkill. I made a simple patch to trace the issue:

func (s Schema) Prepare(......

            if found && (!oFound || !reflect.DeepEqual(v, oValue)) {
                fmt.Printf("field=%s type=%T value=%#v  otype=%T ovalue=%#v \n", field, value, value, oValue, oValue)
                changes[field] = value
            }

Which gives following trace:

field=appointmentDT type=string value="2018-01-05T18:00:00+02:00"  otype=time.Time ovalue=time.Time{wall:0x0, ext:63650764800, loc:(*time.Location)(0xb47380)} 
field=serviceId type=string value="5a4d63fe41095f105dcd49b6"  otype=bson.ObjectId ovalue="ZMc\xfeA\t_\x10]\xcdI\xb6" 
field=id type=string value="5a4d69264e2127a7118b4c51"  otype=bson.ObjectId ovalue="ZMi&N!'\xa7\x11\x8bLQ" 
field=createdDT type=string value="2018-01-04T01:37:18.079+02:00"  otype=time.Time ovalue=time.Time{wall:0x4b571c0, ext:63650619438, loc:(*time.Location)(0xb47380)} 

This indicates that for some types, original values are in DB format, but incoming are in text format. Later in func (s Schema) Validate(...... conversion is made, but this is too late.

Can I propose following change:

            v, err := def.Validator.Validate(value)
            if err != nil {
                log.Panic("Cannot validate")
            }
            if found && (!oFound || !reflect.DeepEqual(v, oValue)) {
                changes[field] = v
            }

Maybe we can refactor Prepare to propagate that error instead of panic.

[rs]

Did you remove the code I describe the other day which copy read only values into the base? It’s to workaround this exact issue.

[Dragomir-Ivanov]

Hmm, where is that description? I didn't see anything on GitHub, or on Slack. Also, it is not just about Read-only values, although rest-layer explicitly errors on these. Implicitly all fields may be affected, and just considered as updates, calling hooks, etc.

[Dragomir-Ivanov]

Also some more explanation about non-read-only values. Using JSON-Patch protocol RFC6902, I use original values, apply the patch on those, and then get the new value. Doing so, I have whole bunch of fields that are actually the original ones, residing in the "new" values, that need to be discarded as updates, and need to go to base.

In the example above "appointmentDT" non-read-only field, has the same value, just in different formats. time.Time in original values, and string in the new values. This is considered as update that will potentially trigger a hook. This is quite a popular use-case. GET-ing values for a form, updating some fields, then PUT-ing the full form back.

[Dragomir-Ivanov]

Okay, looking more at the problem, it seems that Mongo is storing some types in its internal types such as Date and ObjectId that may be need to be converted to string before supplying to original. What you think @rs?

[rs]

I was refering to a different part of the code related to read only but interacting with lookup fields. Ignore my comment.

Yes, the storer engine should guarantee that in == out. We may add that to the doc of the interface of not already.

[Dragomir-Ivanov]

Okay, I think with storer that this is already the case. And it seems a bit awkward we are deciding if a field is update or not, even before we validate it. That is why I think my initial proposal is correct, and we actually call FieldValidator.Validate in schema.Prepare, and we obtain that "internal" type for the field, then compare that with deepEqual to the original one. After that schema.Validate, which calls again FieldValidator.Validate on the fields will get already validated fields, and Validator method will cost us just a type check. Here is example from time.go:

func (v Time) parse(value interface{}) (interface{}, error) {
    if s, ok := value.(string); ok {
        for _, layout := range v.layouts {
            if t, err := time.Parse(layout, s); err == nil {
                value = t
                break
            }
        }
    }
    if _, ok := value.(time.Time); !ok {
        return nil, errors.New("not a time")
    }
    return value, nil
}

If we supply time.Time as value parameter, it will not do the conversion/parsing again.

[smyrman]

Read-only fields in the current schema package are handled here (for PATCH):

• https://github.com/rs/rest-layer/blob/c3ebffd8807d4a5b2a062173729b7cc951984267/rest/method_item_patch.go#L87

and for PUT:

• https://github.com/rs/rest-layer/blob/c3ebffd8807d4a5b2a062173729b7cc951984267/rest/method_item_put.go#L64

But for your case they are not working?

... it seems a bit awkward we are deciding if a field is update or not, even before we validate it. That is why I think my initial proposal is correct, and we actually call FieldValidator.Validate in schema.Prepare.

I have been considering if it's possible to get rid of Prepare all together... It's early stages, but in https://github.com/searis/schema/blob/33e73d98b639bd7bb0f1ac3d883cb99d67b9fcfb/schema.go#L9 I have been toying with a possibly design for allowing to ignore Read-Only checks to be omitted without a separate Prepare call. Ignoring ReadOnly would then probably also be accessible to hooks. Don't know if the design will really work yet, and it may change substantially. This is just one evenings worth of work.

[Dragomir-Ivanov]

As I see it, the code snippets above will work only for id field, because it is the only one residing in the resource path. Other Read-only fields will work only if you don't PUT them at all, however if you PUT them with the old value, rest-layer will raise an error, where I don't think this is the correct behavior.

Now I am experience this issue, because the way JSON-Patch works, basically I am updating the whole resource, along with any read-only fields in it. However this might happen if someone tries to PUT an HTTP form, without stripping read-only fields first.

Also as I stated, this affects not-changed, non-read-only fields as well. They go to changes instead of base.