Usually you would first start by adding new configuration options... meaning, possibly to "libnm-core/nm-setting-wireless.c" (or if it is distinct enough from Wi-Fi a separate "libnm-core/nm-setting-wifi-direct.c" -- but probably not). From there, you would add support to nmcli so that you can create such configurations for testing.
actually, as this is even a large thing, it's not only about new configuration options, but designing/implementing a D-Bus API exposed by NM to expose wifi-direct functionality
maybe the first thing is to implement the D-Bus API server side... or at least, have a plan how that API would look like
maybe just design the server-side API and how it will look like. It may be good to have it is close to connman's API as possible (so that porting between the two implementations is easier). A good start would be to understand how connman exposed Wifi direct and how that API could look like in NM.
dcbw wrote about that already in https://mail.gnome.org/archives/networkmanager-list/2015-January/msg00018.html
it would be good if the dbus API would identical to connman's (like consolekit and logind have the same inhibitor API)... but probably that is not fully possible (see dcbw's email)
Comment from Thomas Haller on IRC: