Need to Port to Python3

[kmitch95120]

I just spent an unreasonable amount of time getting staticDHPCd running on an Ubuntu 20.04 server and I'd have much rather spent that time helping to port this project to Python3.

What will it take to get this going?

[flan]

I don't anticipate that the porting process itself would be terribly hard (and some parts of libpydhcpserver could be simplified by use of the ipaddress module), but the barrier, beyond time, is testing. I don't have a particularly large network with which to do validation or many systems to which I could potentially introduce chaos, so I'd need to delegate validation to others and the rate of anyone passing back even a "yeah, this works" has been pretty close to zero.

The largest technical hurdle is probably custom modules, since the import semantics changed between Python 2 and 3 and it'll probably take some experimentation to get the search-paths lined up properly so it feels intuitive to use, plus the lack of backwards-compatibility that it'll bring.

[kmitch95120]

I'd suggest we create a 2.0.0 branch for Python2 and also a 3.x.x. branch that requires Python3. It's convenient that the staticDHCPd version numbers align with the major Python versions. Is that a sign? As for testing, I'm signing up. As for others not reporting back, people have a tendency not to say anything when things work. They only complain when they don't.

As for backwards compatibility, I'd suggest we keep the 2.0.0 branch for those who wish to use Python2 (at least until they can't get it installed anymore) and also offer some 3.x.x conversion guidance for those with custom modules.

Are you still using your redhamsterx email address? I have that from many years ago when we worked on a Dell PowerEdge IDRAC dhcp issue.

[flan]

That's pretty much what I was planning to do when the time eventually came to do the Python 3 port.

I'll try to get started on it this weekend, but I'm not sure how much time I'll be able to dedicate to it on a regular basis.

Given my recollection of the code, it shouldn't take too long to have something that's testing-ready prepared, assuming a 1:1 conversion without using any new Python 3 features, but it will take about as long to enumerate everything that needs to be tested so it can be checked off.

[flan]

And yeah, any e-mail addresses you may have for me are still usable. Unless you need to keep some information confidential, though, I'm fine with keeping the discussion public, in this thread or others.

[flan]

Work on the 3.0.x branch has started. I think I got everything except potential byte/str conflicts addressed in libpydhcpserver today and I'll chip away at staticDHCPd over the week. I'm thinking I'll be able to do some preliminary testing next weekend, but I don't feel comfortable committing to that target at this time.

[kmitch95120]

Great News! I've started defining test cases and I plan to implement them using scapy. I've also asked one of our support engineers for help in creating a stress test environment using our new workload generator engine called StackPlay. We'll start with 2500 clients and go from there.

[flan]

I didn't get as far as testing, since it was a 96-hour work-week, but I might be able to run it in some capacity tomorrow. TODO: revise extensions, dhcp.py, and config.py, the latter of which will be the most difficult part because of the import semantics; the others are almost all just logging syntax changes and exception-handling.

I think the testing list near the top of README.md covers everything that needs to be verified before reaching a release-candidate state.

[flan]

It's now in an "it runs" state and the web interface displays logs, so I think, syntactically, the code is Python3-compliant.

But I'm hitting str/bytes mismatches in all the places I expected to encounter them, so I'd recommend against trying it right now unless you want to send a pull-request with a bunch of .decode('utf-8')s. I'll probably have time to get it processing packets on Wednesday, if not tomorrow, but it's all subject to work-related demands on my time.

After that, it'll be getting extensions working and documenting any changes in configuration flow for migration, then testing can begin in earnest.

[flan]

It's now in a state where the makedebs script runs and produces packages that can serve, renew, and release leases, so core functionality is verified.

I haven't yet touched extensions, so if you have any custom logic that needs to be imported (rather than defined in-line), I'd expect it to fail. Otherwise, config should be identical to 2.0.x and still support 1.0.x flavours as well, provided any site-defined functions in there are updated to Python 3.

If you don't depend on extensions, you could probably start experimenting with it and seeing if you can cross off any of the testing targets. I'd expect there to be a lot of small errors, though. If you can correct any of them, I'll take pull-requests. Otherwise, pointing me in their direction will help because my testing vectors aren't going to focus on databases until all of the runtime datastructures and hooks are validated, but I can probably fix errors in them quickly if they're flagged.

[flan]

Extensions should now work, though you'll have to make minor changes to conf.py. See the "3.0 errata" section on the 3.0.x branch's main page for information.

I'll try to cross off a testing item or two every day, provided I can set up a suitable environment for them.

[kmitch95120]

Good to hear. I'm buried at work this week but should be able to start testing this weekend. I've setup a Proxmox cluster for isolated testing.

[KevinRiordan]

hey all, I am considering using this tool to conduct what is essentially dhcp relay testing. the callback functions and other features seem to meet my needs. I am a little confused regarding the versions though, and need the tool to be python3 compliant. the releases in the README only show versions 1.x and the documentation refers to 2.x and now you guys are talking about 3.x.

cloned the repo and going to try and play around with it. I would be happy to provide feedback as needed. hopefully everything goes smoothly

[kmitch95120]

Hi Kevin,

The 3.0.x branch supports Python 3. After you clone the repo you'll need to switch to the 3.0.x branch.

Let us know if you need a hand.

[flan]

2.x never saw a formal release because I felt uncertain about calling it "ready" with some untested features. They were implemented in response to tickets, but I never received confirmation that they worked -- the other party just went silent at the end, so it ended up dangling.

I'm planning to be more aggressive with calling 3.0 ready when I'm mostly confident stuff won't catch fire, since it's become apparent that people are more likely to use it (and therefore report issues) if they see something in the releases list or don't see "release candidate" for years on end, even when it's probably perfectly stable and production-ready.

I would still greatly appreciate confirmation that anything in the to-be-tested section at the top of the README works for you, though. And any 2.0 documentation and discussion should still apply to 3.0. Aside from the caching-related issues that are still open, which I'm working on this weekend, I'm intending for it to be as close to identical as possible, just targeting Python 3.5+ instead of 2.3+.

[KevinRiordan]

Hi Kevin,

The 3.0.x branch supports Python 3. After you clone the repo you'll need to switch to the 3.0.x branch.

Let us know if you need a hand.

Thank you for that, I was able to clone and then checkout the 3.0 branch. I hope this tool will meet my needs and I will be happy to provide feedback as needed considering the amount of time this saves me from writing a dhcp server from scratch.

I would like to have a test suite communicate with the dhcp server to run certain tests, including negative tests. I have been reading over the scripting documentation, and the callback section, but am a little confused on how to begin an implementation. is there an existing framework for how to dynamically control the server to send messages as desired. for example it would be nice if i could tell server to send an offer to all clients. or to send two acks, or things of that nature. I happy to build the tools as needed just not sure where to start exactly. any help is much appreciated an I will do my best to provide the most detailed feedback as possible.

[flan]

The server itself is intended to be well-behaved according to https://www.ietf.org/rfc/rfc2131.txt and the hooks exposed aren't intended to make it easy to break that expectation.

What you might want to do is use libpydhcpserver directly, since that gives you access to all the packet internals and the UDP interfaces you'll need to send and receive, but abstracted to a fairly accessible level. staticDHCPd is, at its core, a subclass of libpydhcpserver.dhcp.DHCPServer that adds the MAC-to-IP logic, callbacks, and other features that make its role a bit special in the server ecosystem. See https://github.com/flan/staticdhcpd/blob/7497e6b4f4d10219317588f64e38a6e9919d9dc0/staticDHCPd/staticdhcpdlib/dhcp.py#L367 and https://github.com/flan/staticdhcpd/blob/7497e6b4f4d10219317588f64e38a6e9919d9dc0/staticDHCPd/staticdhcpdlib/dhcp.py#L811 for the relevant sections for implementation; if you just want to manipulate a network, you should only need a small slice of what's implemented there.

I'd start by copying that file and just deleting the pieces you don't need until it starts to make sense, then build up the logic you actually want.

Also, since this will probably diverge quite a bit from the 3.0 port of staticDHCPd, it would probably be best to start a new issue to keep things focused. I'm reasonably certain that the port of libpydhcpserver to Python3 is stable and complete at this point and I don't expect its API to radically change... ever. DHCPv4 is a pretty fixed quantity at this point.

[kmitch95120]

For negative test cases, you may want to look at https://scapy.net/

[flan]

I've added a minimal example showing how to use libpydhcpserver directly with well-behaved clients with just enough functionality to do useful work: https://github.com/flan/staticdhcpd/blob/3.0.x/libpydhcpserver/examples/hardcoded_server.py

I'd absolutely not suggest anyone take that into a production context, but for poking at a network or creating another server based on the library, it'll be a decent starting point.

If you need more guidance, Kevin, please do create another issue.

[andrew-azarov]

I've received this error when trying to run a Py3 version:

2021-05-30 13:07:06,922 : CRITICAL : Unable to handle DISCOVER from  :
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/staticdhcpdlib/dhcp.py", line 361, in wrappedHandler
    f(self, wrapper)

  File "/usr/local/lib/python3.6/dist-packages/staticdhcpdlib/dhcp.py", line 473, in _handleDHCPDiscover
    if wrapper.loadDHCPPacket(definition):

  File "/usr/local/lib/python3.6/dist-packages/staticdhcpdlib/dhcp.py", line 303, in loadDHCPPacket
    self._loadDHCPPacket(definition, inform)

  File "/usr/local/lib/python3.6/dist-packages/staticdhcpdlib/dhcp.py", line 283, in _loadDHCPPacket
    self.packet.setOption(12, definition.hostname)

  File "/usr/local/lib/python3.6/dist-packages/libpydhcpserver/dhcp_types/packet.py", line 632, in setOption
    value = self._extractList(value, option=option)

  File "/usr/local/lib/python3.6/dist-packages/libpydhcpserver/dhcp_types/packet.py", line 445, in _extractList
    return self._serialiseOptionValue(option, value)

  File "/usr/local/lib/python3.6/dist-packages/libpydhcpserver/dhcp_types/packet.py", line 390, in _serialiseOptionValue
    return _FORMAT_CONVERSION_SERIAL[type](value)

  File "/usr/local/lib/python3.6/dist-packages/libpydhcpserver/dhcp_types/conversion.py", line 162, in strToList
    return [ord(c) for c in s.encode('utf-8')]

  File "/usr/local/lib/python3.6/dist-packages/libpydhcpserver/dhcp_types/conversion.py", line 162, in <listcomp>
    return [ord(c) for c in s.encode('utf-8')]

TypeError: ord() expected string of length 1, but int found

Solution is to replace return [ord(c) for c in s.encode('utf-8')] with return list(s.encode('utf-8'))

[flan]

I was definitely expecting to encounter a bunch of str/bytes things like this as the migration proceeded.

Thanks for reporting this one; it's been addressed in the most recent commit, though I needed to make the implementation a bit more complex to handle both unicode and byte-strings as input.

Other functions were updated, too, to mitigate potential issues like this in the future.

And fixes related to what was formerly integer-division.

[flan]

The cycle of having virtually no free time has continued, unfortunately, but I'll be rolling this system out for a development and testing environment at work, so I'll finally be able to close the remaining gaps and make Python 3 support, alongside 3.0.0, official in a few weeks, once infrastructure cutover happens.

A number of open concerns have been tested and removed from the to-do list in preparation.

The years-long beta is finally ending.

[fwiessner]

Today, we migrated to staticDHCPd 3 on our prod environment, no problems so far.