Request.form data dict... Could we modify the structure?

[Lewiscowles1986]

using the following html form

<h1>Debug Form</h1>
<form action="/debug" method="POST">
<input name="item[1][name]" value="" />
<input name="item[2][name]" value="" />
<button type="submit">test</button>
</form>

to the following flask application method / view

@app.route('/debug', methods=["GET","POST","PUT","PATCH","DELETE"])
def debug_info():
    return jsonify({"data":request.data,"formdata":request.form})

I get

{
  "data": "", 
  "formdata": {
    "item[1][name]": "fdhnnhdsfsdslkkl", 
    "item[2][name]": "djdfhdjfh"
  }
}

I expected to see

{
  "data": "", 
  "formdata": {
    "item":[
      {
         "name": "fdhnnhdsfsdslkkl"
       },
       { 
         "name": "djdfhdjfh"
       }
    ]
  }
}

The first and actual output requires further, more explicit parsing, from a sub-optimal data structure to achieve a nice dictionary; but I cannot see how the flat structure of the request.form object could be helping anyone...

[davidism]

You've given the inputs two different names, so of course you get two different values. Give them the same name, and you get a list of values. However, you cannot (basically by definition) get a nested structure from an HTML form, no matter what special names you try to give the inputs. HTML forms are flat.

[Lewiscowles1986]

@davidism I think you may be confused mate. If I give two elements the same name there would be no identifier to link grouped elements. so...

using the following html form

<h1>Debug Form</h1>
<form action="/debug" method="POST">
<p>
  <input name="item[1][name]" value="" />
  <input name="item[1][price]" value="" />
</p><p>
  <input name="item[2][name]" value="" />
  <input name="item[2][price]" value="" />
</p>
<button type="submit">test</button>
</form>

I would expect

{
  "data": "", 
  "formdata": {
    "item":[
      {
         "name": "fdhnnhdsfsdslkkl",
         "price":"3.99"
       },
       { 
         "name": "djdfhdjfh",
         "price":"21.99"
       }
    ]
  }
}

[RonnyPfannschmidt]

the name is clearly a sting, there is no standard to tell how to mp that from/to a nested structure

[Lewiscowles1986]

for now I am going to try using a helper to facilitate, but it means I will need to know how many item's are being passed through

def parse_nstring( form, name, patht ):
    try:
        key = "".join( [ name, '[', "][".join( patht ) , ']' ] )
        return form[ key ]
    except:
        return None

it was modeled from the following structure using python

req = {"form":{"item[1][name]":"lewis","item[2][name]":"peter"}}
parse_nstring( req['form'], "item", ["1","name"] )
parse_nstring( req['form'], "item", ["2","name"] )
parse_nstring( req['form'], "item", ["2","age"] )

also @RonnyPfannschmidt "there is no standard to tell how to mp that from/to a nested structure"... I've never actually had to map it before, and this is not my first rodeo

this covers passing formdata as dicts

[RonnyPfannschmidt]

there are some libraries that map such names to nested form validation (the path schemes may differ)

[Lewiscowles1986]

could you give me some examples please @RonnyPfannschmidt Google is not helping in the context of flask or libraries for this (without form abstraction like in WTForms)

[RonnyPfannschmidt]

form abstraction is exactly what i meant

[ThiefMaster]

FYI, automated conversion from specially formatted field names to nested structures as a default on the framework level is a terrible idea. It is not up to the client to decide whether a variable on the server side should be a plain string or a list/dict.

[mattupstate]

"Just use JSON"™

[Lewiscowles1986]

@RonnyPfannschmidt I suspected so, the reason for not wanting form abstraction is that it still deals with known fields... Not good for rapid prototyping.

@ThiefMaster The client is not "deciding", it is submitting data that has a clear form and structure, which I think would be nice for Flask to play along with.

@mattupstate :+1: :smile: I do that in another flask example, where I literally send JSON to the server, but this is for something I want to try out using HTML forms, where generating JSON is not then a client-side problem. I have a functional demo in PHP, but meh, I prefer to use Flask as this is an experiment in rapid prototyping with Flask.

N.B. Thank you all for your help, I am not ready to give up on my concept yet, but I do appreciate your advice, and time, and patience.

[Lewiscowles1986]

_Update_

Went for a walk in the park with my wife, came home, answered comments, thought of another idea...

import re

'''
Parses depth, encoded names into a JSON'ish dictionary structure
''' 
def parse_to_dict_val( key, value, dict={} ):
    patt = re.compile(r'(?P<name>.*?)[\[](?P<key>.*?)[\]](?P<remaining>.*?)$')
    matcher = patt.match( key )
    tmp = ( matcher.groupdict() if not matcher == None else { "name": key, "remaining": '' } )
    if tmp[ 'remaining' ] == '':
        dict[ tmp[ 'name' ] ] = value
    else: # looks like we have more to do
        fwdDict = ( dict[ tmp[ 'name' ] ] if tmp[ 'name' ] in dict else {} )
        tmp2 = parse_to_dict_val( tmp[ 'key' ], value, fwdDict )
        dict[ tmp[ 'name' ] ] = tmp2
    return dict

'''
Parses dictionary for encoded keys signifying depth
'''
def parse_to_dict_vals( dictin ):
    dictout = {}
    for key, value in dictin.items():
        parse_to_dict_val( key, value, dictout )
    return dictout

req = {"form":{"item[1][name]":"bob","item[2][name]":"dylan","nameflat":"jimmy"}}

parse_to_dict_vals( req[ 'form' ] )

it doesn't work yet, but it might get me and anyone else like me to a happy end-result

:blush: Looks like that code is solid. Again this is only for rapid prototyping, but I think, even if included as a utility, the parse_to_dict_vals function is a nice way of being able to model data structures in-browser.

[RonnyPfannschmidt]

again, there is no universal standard for such a structure, for structured data you should pass json from the browser, not form-encode with a "interesting" naming scheme and hope for the best

[Lewiscowles1986]

@RonnyPfannschmidt with the greatest of respect, no standard exists, before it exists... If we simply threw our hands up in the air and used something that existed all the time, but was not a good fit, every time we wanted to do something new, we would loose out on a lot, and would likely still be feeding punch cards into a mainframe we have to book time on.

There are other languages that handle nested form data as I have mentioned, but I would like to use Flask and python. I do accept, it may not be for everyone, and may not be for the core of Flask, it seems others have strong opinions on this, but this is valid code...

Anyway I have now completed the rapid prototyping hierarchical form data functions.

Objections and Answers I can think of

• If someone throws data your app is not meant to be dealing with, you should not be using it! This is for working with known data, without abstracting HTML Forms in your app, not sending data directly to a database or anything insane. That is it; and is all I will be using it for...
• If someone throws too much data (for recursion bomb attacks on nested structures etc), you should be able to easily add a fourth param as a counter, or reject the request if too large... JSON is likely as attackable as this format... A Simple fix is put nginx, or similar in front of the application nodes, and adjust their settings to reject the request before it reaches the app...
• If you are worried about people sending the wrong data-types to properties, add checking to the properties you wish to check before operating on them...
• If you would rather write more code to abstract a form, and / or use front-end to transform the data into JSON; be my guest! The front-end is not exactly the best place to be building payloads to pass to the back-end IMHO; and it is certainly not "rapid development", or "lean" to have to abstract every form, of every part of your app. Instead I would focus on building validation into the models, and classes you use themselves, so anything that writes to them follows a structured and mediated approach to modifying state (which is all WTForms achieves). You will spend less time building front-ends, and clients to an app with consistent data handling at lower-levels, and less time re-writing or dealing with edge-cases; pushing real end-user benefits, as you can focus on adding more features...

anyway https://gist.github.com/Lewiscowles1986/380425897b456a3f0d5b use-it, don't use it, I GPL'd it, so it's free free

[RonnyPfannschmidt]

@Lewiscowles1986 its entirely valid to put this into a opinionated extension/custom lib and use it for the prototyping mechanism you came up

this is one of the reasons why flask does not come with a standard tool for exact form handling, flexible extensions and libraries leave the experienced users so many more ways choose what they need for their use-cases

this is really important, since many use-cases and design choices can look wrong from a different and/or uninformed perspective.

however since json and fully blown form validation already have well working solutions for this as well, having a custom semi-standard format for it seems overbearing and a maintenance burden at the framework level i don't see flask or werkzeug as place for this

by now its a well accepted idea being well-accepting of "anything" instead of being strict leads to a plaque of semi-compatible messy implementations (the best example for such a mess is webdav and the related standards on top (caldav/carddav sync)

while not being strict eases initial development and rapid experimentation it has immense costs for further evolution and inter-op

the loss that gradually incurs over time due to ever growing implementation complexity by far out-weights the potential for short term gains

my personal opinion these days is that anything that does not use well and strictly defined hierarchical data formats for prototyping of hierarchical structures will generate and immense maintenance cost

as for rapid prototyping - my own experience is that automatically generating forms from annotated models and enriching those with custom behavior as the need grows tends, can easily win over prototyping from html

however in such cases the auto-generated form handling already completely handles the un-flattening of the form as well as the rendering of the elements to html

so from my point of view the structure you want to advocate is an intermediate development step that can easily be replaced by already existing well defined structural formats as well as already existing structure oriented unflattering based on more integrated data validation

with this i also end my day, its quite late

[Lewiscowles1986]

@RonnyPfannschmidt :+1: I am again, as I have already said, and now reiterating again that I am not advocating accepting "anything", but rather making it easier to accept form data in a structured format without tooling lots of code to facilitate.

being well-accepting of "anything" instead of being strict leads to a plaque of semi-compatible messy implementations

Have a great night, I'm going to close this now anyway :wink:

[RonnyPfannschmidt]

@Lewiscowles1986 by chance i stumbled upon https://darobin.github.io/formic/specs/json/ due to work this morning,

so there already is a in browser standard to turn the structure you roughly describe into properly formed json documents for submission, and additionally it handles many edge-cases your code snipped doesn't even think of

[Lewiscowles1986]

@RonnyPfannschmidt thanks for the link! You know that is an unofficial draft sample that is inconsistent, unsupported (at least as far as my testing goes with Firefox and Chrome), and the only thing it adds other than inconsistency, I can see is file support, which due to chunked uploads etc should be abstracted in-browser where supported.

Yesterday everyone was so...

universal standard

Today seems different. Why?

EXAMPLE 6: Such Deep was more than a little worrying

• that it would generate inconsistent output of tuple or object, depending upon non-numeric key, which is something my code does not have... keys where defined, shall be explicit
  • so while I would not at present support "people[][name]"="bob", there is very good reasoning for this. Given no key to link the [name] to specific siblings, the implementation would be forced to work on the understanding that
  • the browser encodes, and transmits all form data in order of definition
  • the front-end would then be structurally limited in this way
  • this would also predicate that flask guarantees MultiDict & dict comply with order added for all keys, and decodes keys in order of sending.

Call me crazy, but that doesn't seem nearly as useful as requiring explicit keying, and unique naming for all elements, not allowing sending of multiple elements without specified key for all, and overwriting of duplicate keys with latest parsed value (which cannot be guaranteed due to dict, but, should not come up in a good front-end implementation anyway).

I think the confusion might be that my code is not for generating JSON. It generates a dictionary... I output JSON from a view, as there is standard tooling for this, and as with all innovation, I don't want to be completely off the reservation, and loose all ability to measure, so I can view the structure in a simple, clear format.

Again, this is not going into a production web-app or being sold to anyone, this is an experiment in rapid prototyping in flask, and I have closed the issue as I see it now more as a could have for applications, but it should not require additional W3C standards, or any additional front-end work, the idea is to keep lean on the front-end, and allow the back-end more logical format to work with...

[RempelOliveira]

@RonnyPfannschmidt thanks for the link! You know that is an unofficial draft sample that is inconsistent, unsupported (at least as far as my testing goes with Firefox and Chrome), and the only thing it adds other than inconsistency, I can see is file support, which due to chunked uploads etc should be abstracted in-browser where supported.

Yesterday everyone was so...

universal standard

Today seems different. Why?

EXAMPLE 6: Such Deep was more than a little worrying

• that it would generate inconsistent output of tuple or object, depending upon non-numeric key, which is something my code does not have... keys where defined, shall be explicit

  • so while I would not at present support "people[][name]"="bob", there is very good reasoning for this. Given no key to link the [name] to specific siblings, the implementation would be forced to work on the understanding that

  • the browser encodes, and transmits all form data in order of definition

  • the front-end would then be structurally limited in this way

  • this would also predicate that flask guarantees MultiDict & dict comply with order added for all keys, and decodes keys in order of sending.

Call me crazy, but that doesn't seem nearly as useful as requiring explicit keying, and unique naming for all elements, not allowing sending of multiple elements without specified key for all, and overwriting of duplicate keys with latest parsed value (which cannot be guaranteed due to dict, but, should not come up in a good front-end implementation anyway).

I think the confusion might be that my code is not for generating JSON. It generates a dictionary... I output JSON from a view, as there is standard tooling for this, and as with all innovation, I don't want to be completely off the reservation, and loose all ability to measure, so I can view the structure in a simple, clear format.

Again, this is not going into a production web-app or being sold to anyone, this is an experiment in rapid prototyping in flask, and I have closed the issue as I see it now more as a could have for applications, but it should not require additional W3C standards, or any additional front-end work, the idea is to keep lean on the front-end, and allow the back-end more logical format to work with...

I also don’t understand why flask is not able to correctly analyze the form-data array structures of the form, php, .net, ruby ​​and other development languages ​​do