Adding Python extensions support for Windows

[muffins]

This adds in an implementation of TPipe for windows, which allows us to make use of Python extensions on the Windows platform.

Some samples of the extensions in action. First we startup osquery:

:\Users\Nick\work\repos\osquery [win-python-auto-ext]
λ  .\build\windows10\osquery\RelWithDebInfo\osqueryd.exe -S --flagfile=C:\Users\Nick\work\configs\osquery_extensions\osquery.flags --verbose

Then we start up our extension:

C:\Users\Nick\work\repos\osquery-python [osquery-python-windows-port +1 ~0 -0 !]
λ  python .\foobar_table.ext --socket \\.\pipe\shell.em --verbose

We see the extension connect to our shell:

I0131 16:07:10.660527  4272 init.cpp:380] osquery initialized [version=3.0.0-17-g10c0c60b]
I0131 16:07:10.663009  4272 rocksdb.cpp:132] Opening RocksDB handle: C:\Users\Nick\work\configs\osquery_extensions\osquery.db
I0131 16:07:10.725503 22132 interface.cpp:338] Extension manager service starting: \\.\pipe\shell.em
I0131 16:07:10.741130 21252 interface.cpp:89] Thrift message: TPipe ::GetOverlappedResult errored GLE=errno = 109
I0131 16:07:10.741130 21252 interface.cpp:89] Thrift message: TConnectedClient died: TPipe: GetOverlappedResult failed
I0131 16:07:10.741130  5972 interface.cpp:89] Thrift message: TPipe ::GetOverlappedResult errored GLE=errno = 109
I0131 16:07:10.741130  5972 interface.cpp:89] Thrift message: TConnectedClient died: TPipe: GetOverlappedResult failed
Using a I0131 16:07:11.811651 25176 events.cpp:746] Starting event publisher run loop: windows_event_log
virtual database. Need help, type '.help'
osquery> I0131 16:07:12.191326  8460 interface.cpp:141] Registering extension (foobar_table, 29953, version=1.0.0, sdk=1.8.0)
I0131 16:07:12.191326  8460 registry.cpp:351] Extension 29953 registered table plugin foobar

We then query the extension:

osquery> select * from foobar;
+-----+-----+
| foo | baz |
+-----+-----+
| bar | baz |
| bar | baz |
+-----+-----+

So far I have tested extensions being autloaded with osquery, which currently works from both command line and running as a system service. I tested having a python extension that both provides data to, and queries data from osquery, and this didn't seem to work. I'm still digging into what's happening to prevent bidirectional communication, but throwing this up to get the review process started.

[muffins]

Huh, not entirely sure what I'm doing to screw up tests, however funny thing - tests pass on Windows :D

C:\Users\Nick\work\repos\osquery-python [osquery-python-windows-port]
λ  python setup.py test
running test
running egg_info
writing requirements to osquery.egg-info\requires.txt
writing osquery.egg-info\PKG-INFO
writing top-level names to osquery.egg-info\top_level.txt
writing dependency_links to osquery.egg-info\dependency_links.txt
reading manifest file 'osquery.egg-info\SOURCES.txt'
writing manifest file 'osquery.egg-info\SOURCES.txt'
running build_ext
test_simple_call (tests.test_config_plugin.TestConfigPlugin)
Tests for the call method of osquery.TablePlugin ... ok
test_simple_call (tests.test_logger_plugin.TestLoggerPlugin)
Tests for the call method of osquery.TablePlugin ... ok
test_plugin_inheritance (tests.test_plugin.TestBasePlugin)
Test that an object derived from BasePlugin works properly ... ok
test_singleton_creation (tests.test_singleton.TestSingleton)
Test that two singletons are the same object ... ok
test_plugin_was_registered (tests.test_table_plugin.TestTablePlugin)
Tests to ensure that a plugin was registered ... ok
test_routes_are_correct (tests.test_table_plugin.TestTablePlugin)
Tests to ensure that a plugins routes are correct ... ok
test_simple_call (tests.test_table_plugin.TestTablePlugin)
Tests for the call method of osquery.TablePlugin ... ok

----------------------------------------------------------------------
Ran 7 tests in 0.032s

OK

[muffins]

Looks like the only failing test is due to Python 2.6 no longer being supported:

isort requires Python '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*' but the running Python is 2.6.9
The command "pip install -r requirements.txt" failed and exited with 1 during .
Your build has been stopped.

[muffins]

With the most recent commit I've verified this does not impact python extensions on posix platforms.

[ryanheffernan]

I found a few bugs, see above comments. With those fixed I'd be good to merge, was able to get this working successfully once I made the changes locally.

[ryanheffernan]

I'm good with these changes, code looks good and testing proved successful. I am seeing some stability issues but need further testing and am OK with this being shipped in current state.

[ryanheffernan]

One note, which i can file as a seperate issue if you prefer, is that when using osqueryi to call an extension table, it works the first time but then doesn't work until i kill and restart osqueryi.

After running the select * from an extension table in osqueryi, after the first time all subsequent runs just return nothing until i restart a new osqueryi process.

[ryanheffernan]

BTW some of the stability issues I saw look similar to #3954 on osquery, so this likely isn't related to this diff but a generic windows extensions issue.

[theopolis]

:+1:

[theopolis]

We can remove build for 2.6 in a follow up PR, do not let that stop you from merging this.

[muffins]

I'm going to move forward with landing this as in it's current form this is working on Windows with osquery running as a system level service, and a configuration that queries against a python extensions table. Any remaining issues that folks have for this can be filed as follow up issues and I'll deal with them :)

[muffins]

@sudhirsingh20 you’ll wanna check what version of the osquery python module you’ve got installed, as that looks like the old logic. Ensure you’re running off of 3.0.2, and if you have more problems feel free to open an issue