Redundancy (clustering) support

[rlagmlah]

Hello. TechnitiumDNS is often unresponsive. The cause is unknown, but usually when run for more than 3 days in a row. This issue seems to have no problems during the daytime if I restart TechnitiumDNS every night, but with a low probability (2% in my case) there was a case where TechnitiumDNS stopped working. To solve this, 2 TechnitiumDNS were executed respectively, and 2 TechnitiumDNS were registered in the DHCP server, but it is difficult to manage the settings and statistics because each DNS server has independent configuration files. I think it will be very helpful for stability if it supports clustering officially. thank.

[ShreyasZare]

While this issue was added in April 2020, the project lacked many core features like DNSSEC, EDNS to name a few and had several issues with handling large throughput. There are a few ISPs that started using the DNS project and it was important to fix performance issues and bugs that were stalling the DNS server in production. These things were much more important to implement. Which is why clustering feature was not prioritized since it was not the right time to implement it.

Another reason for not prioritizing clustering feature was that it could be done manually too. Sure, it would need a lot of action to be taken to maintain multiple instances but it is still achievable and I am managing a couple of "clusters" in similar way.

Clustering was later planned a few times to implement it but due to lack of time and requests for features where were popular got implemented instead. It was also planned to implement Catalog Zones for clustering which was at draft stage in IETF during that time. It was also planned to first get the DHCP server major update done with support for DHCPv6 and include it too in the clustering feature. However, clustering feature got requested frequently and around Oct 2023 it was decided to delay DHCP server major update and move ahead with clustering support only for DNS. Its unfortunate that @computerkoenig thinks this was "lies" based on some comments and does not see the circumstances.

This project is being developed just by one person (me) and that too after I can find some time after my full time job. So, when I know that I have 4 hrs of time, I chose to fix several issues that get reported or add small feature/options so as to utilize the time to cover as many things as possible. There is also attempt to release update every month or two so that any bug fixes that were reported get fixed for everyone.

That's kind of what appears to be the problem. They're hesitant to let people work on the core of the project because "core features require deep understanding of DNS concepts".

This project does requires having a level of expertise in DNS concepts. It would actually take more time to manage various discussions and review PR than to get core features implemented directly. Core features tend to require changing a lot of existing implementation and thus understanding of the entire codebase is important. Every PR that gets added adds another task for me to review the code, refactor it, make sure existing features do not break and do a lot of testing. With clustering feature, its even more important to get the design right at initial stage. It would become too difficult to fix any design issues once the feature is deployed.

powerdns for example. dns clustering is the most common thing dns servers are able to. even this ... windows dns server.

Its actually nice that this project is being compared with other DNS servers. Important thing here is that those projects have entire companies behind them with lot of resources.

Anyways, let me thank everyone for being patient here. Current status of clustering feature is in design stage where I am studying catalog zones and how its implemented in BIND along with ways to see how proprietary conditional forwarder zone be implemented with it. There is also plan to include support to "upgrade" a secondary node in cluster to "primary" when primary node crashes for some reasons. This is a bit tricky since DNSSEC signed primary zones only have access to private keys currently and a safe way to share the keys to secondary nodes needs to be designed.

Meanwhile, there are a few bug fixes and some minor features that is planned to be released in next minor update that is upcoming in couple of weeks.

[computerkoenig]

@ShreyasZare okay. most things I understand.

But

1. in my opinion is everyone a lier, who tells other people anything that will not happen. If there are complications or more importent things, you need to correct what you said or you're a lier. I always inform everyone I told that something will happen, which was not right/possible/...

2. What the f... does this fucking DHCP feature do inside a DNS Server? Inside an really fast and load tollerant DNS deamon? Do you want to do everything, but nothing perfect?

3. If you would be able to communicate and talk to all the people wich are waiting and interrested, maybe there would be someone who is good enough (in your eyes) to just help you with this f***ing cluster feature. its really not as difficult, as a Mars mission. But maybe is this part of the reason why you are mostly alone working on this project.

Once you are open-minded and grown up enough to let us help, please let us know.

Kind regards, Until a few hours ago, one of your longest admirers

While this issue was added in April 2020, the project lacked many core features like DNSSEC, EDNS to name a few and had several issues with handling large throughput. There are a few ISPs that started using the DNS project and it was important to fix performance issues and bugs that were stalling the DNS server in production. These things were much more important to implement. Which is why clustering feature was not prioritized since it was not the right time to implement it.

Another reason for not prioritizing clustering feature was that it could be done manually too. Sure, it would need a lot of action to be taken to maintain multiple instances but it is still achievable and I am managing a couple of "clusters" in similar way.

Clustering was later planned a few times to implement it but due to lack of time and requests for features where were popular got implemented instead. It was also planned to implement Catalog Zones for clustering which was at draft stage in IETF during that time. It was also planned to first get the DHCP server major update done with support for DHCPv6 and include it too in the clustering feature. However, clustering feature got requested frequently and around Oct 2023 it was decided to delay DHCP server major update and move ahead with clustering support only for DNS. Its unfortunate that @computerkoenig thinks this was "lies" based on some comments and does not see the circumstances.

This project is being developed just by one person (me) and that too after I can find some time after my full time job. So, when I know that I have 4 hrs of time, I chose to fix several issues that get reported or add small feature/options so as to utilize the time to cover as many things as possible. There is also attempt to release update every month or two so that any bug fixes that were reported get fixed for everyone.

That's kind of what appears to be the problem. They're hesitant to let people work on the core of the project because "core features require deep understanding of DNS concepts".

This project does requires having a level of expertise in DNS concepts. It would actually take more time to manage various discussions and review PR than to get core features implemented directly. Core features tend to require changing a lot of existing implementation and thus understanding of the entire codebase is important. Every PR that gets added adds another task for me to review the code, refactor it, make sure existing features do not break and do a lot of testing. With clustering feature, its even more important to get the design right at initial stage. It would become too difficult to fix any design issues once the feature is deployed.

powerdns for example. dns clustering is the most common thing dns servers are able to. even this ... windows dns server.

Its actually nice that this project is being compared with other DNS servers. Important thing here is that those projects have entire companies behind them with lot of resources.

Anyways, let me thank everyone for being patient here. Current status of clustering feature is in design stage where I am studying catalog zones and how its implemented in BIND along with ways to see how proprietary conditional forwarder zone be implemented with it. There is also plan to include support to "upgrade" a secondary node in cluster to "primary" when primary node crashes for some reasons. This is a bit tricky since DNSSEC signed primary zones only have access to private keys currently and a safe way to share the keys to secondary nodes needs to be designed.

Meanwhile, there are a few bug fixes and some minor features that is planned to be released in next minor update that is upcoming in couple of weeks.

[Zorlin]

Christ, be mature. You're being incredibly rude.

Calm down when you're typing.

DHCP is an incredibly useful feature. If you want DNS clustering you can already achieve that with a primary and two secondary DNS instances of Technitium. Just configure a TSIG key and setup a shared zone that replicates between all three. Done. Now quit moaning.

On Fri, 24 May 2024, 16:59 computerkoenig, @.***> wrote:

@ShreyasZare https://github.com/ShreyasZare okay. most things I understand.

But 1. is, in my opinion, everyone a lier, who tells other people anything with will not happen. If there are complications or more importent things, you need to correct what you said or you're a lier. I always inform everyone I told that something will happen, which was not right/possible/...

2.

What the f... does this fucking DHCP feature do inside a DNS Server? Inside an really fast and load tollerant DNS deamon? Do you want to do everything, but nothing perfect? 3.

If you would be able to communicate and talk to all the people wich are waiting and interrested, maybe there would be someone who is good enough (in your eyes) to just help you with this f***ing cluster feature. its really not as difficult, as a Mars mission. But maybe is this part of the reason why you are mostly alone working on this project.

Once you are open-minded and grown up enough to let us help, please let us know.

Kind regards, Until a few hours ago, one of your longest admirers

While this issue was added in April 2020, the project lacked many core features like DNSSEC, EDNS to name a few and had several issues with handling large throughput. There are a few ISPs that started using the DNS project and it was important to fix performance issues and bugs that were stalling the DNS server in production. These things were much more important to implement. Which is why clustering feature was not prioritized since it was not the right time to implement it.

Another reason for not prioritizing clustering feature was that it could be done manually too. Sure, it would need a lot of action to be taken to maintain multiple instances but it is still achievable and I am managing a couple of "clusters" in similar way.

Clustering was later planned a few times to implement it but due to lack of time and requests for features where were popular got implemented instead. It was also planned to implement Catalog Zones https://datatracker.ietf.org/doc/rfc9432/ for clustering which was at draft stage in IETF during that time. It was also planned to first get the DHCP server major update done with support for DHCPv6 and include it too in the clustering feature. However, clustering feature got requested frequently and around Oct 2023 https://github.com/TechnitiumSoftware/DnsServer/issues/134#issuecomment-1787544766 it was decided to delay DHCP server major update and move ahead with clustering support only for DNS. Its unfortunate that @computerkoenig https://github.com/computerkoenig thinks this was "lies" based on some comments and does not see the circumstances.

This project is being developed just by one person (me) and that too after I can find some time after my full time job. So, when I know that I have 4 hrs of time, I chose to fix several issues that get reported or add small feature/options so as to utilize the time to cover as many things as possible. There is also attempt to release update every month or two so that any bug fixes that were reported get fixed for everyone.

That's kind of what appears to be the problem. They're hesitant to let people work on the core of the project because "core features require deep understanding of DNS concepts https://www.reddit.com/r/technitium/comments/1ctqizc/comment/l5ba3jj".

This project does requires having a level of expertise in DNS concepts. It would actually take more time to manage various discussions and review PR than to get core features implemented directly. Core features tend to require changing a lot of existing implementation and thus understanding of the entire codebase is important. Every PR that gets added adds another task for me to review the code, refactor it, make sure existing features do not break and do a lot of testing. With clustering feature, its even more important to get the design right at initial stage. It would become too difficult to fix any design issues once the feature is deployed.

powerdns for example. dns clustering is the most common thing dns servers are able to. even this ... windows dns server.

Its actually nice that this project is being compared with other DNS servers. Important thing here is that those projects have entire companies behind them with lot of resources.

Anyways, let me thank everyone for being patient here. Current status of clustering feature is in design stage where I am studying catalog zones and how its implemented in BIND along with ways to see how proprietary conditional forwarder zone be implemented with it. There is also plan to include support to "upgrade" a secondary node in cluster to "primary" when primary node crashes for some reasons. This is a bit tricky since DNSSEC signed primary zones only have access to private keys currently and a safe way to share the keys to secondary nodes needs to be designed.

Meanwhile, there are a few bug fixes and some minor features that is planned to be released in next minor update that is upcoming in couple of weeks.

— Reply to this email directly, view it on GitHub https://github.com/TechnitiumSoftware/DnsServer/issues/134#issuecomment-2129887100, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKONLEMXCN2XTSTGFTHQRTZD5PXVAVCNFSM4MOZHPLKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJSHE4DQNZRGAYA . You are receiving this because you commented.Message ID: @.***>

[liamfleming26]

Anyone else getting tired of listening to this dude?

If you can't be patient, write the required code yourself and submit a PR. Or fork it and fork off

The suggestion above regarding primary and secondary zones ticks my boxes in terms of redundancy. The bit I'll patiently await is the clustering of conditional forwarder zones.

[computerkoenig]

@Zorlin: Try it in a hyperscaled kuberneetes cluster and share your config if its working and I will stop moaning. sorry that I'm not working with 20 years old systems like others.

...and @liamfleming26 you can try it too. Or fork off :)

[NH-Networks]

@Zorlin: Try it in a hyperscaled kuberneetes cluster and share your config if its working and I will stop moaning. sorry that I'm not working with 20 years old systems like others.

Lol, it will also work in a Kubernetes cluster or docker or LXC.. TSIG has nothing to do with platform you run it on.

So read the manual and set it up.

[computerkoenig]

@NH-Networks (btw: I like you nickname. could you please set it up for me? I will pay you!) It works, yes. but not with automatic scaling, cause of the missinng clustering feature. Or am I to stupid to read the f. manual? If your right, I will say sorry!

...but until now, it only seams like you have no idea what I am talking about. Sorry to all other readers, who are not acting like a godness!

[NH-Networks]

Even with clustering it will not automatically scale.. you can scale it like you want, with as many Secondary zones you like.. you are rights that it is not really clustering, but you can achieve what you want in Kubernetes with multiple instances or just throwing more resources at it.. whatever you like

I'm running 4 docker instances over 2 datacenters.. 1 primary and 3 secondary.. but yes full clustering would be nicer.. but the setup works like a charm

Oh and ..Just please keep respectful, don't forget that this great software is free, bro..

4 instances serve more then 180K clients to give you a idea.

[liamfleming26]

Poorly worded questions get poor answers.

Basic primary and secondary zone setups can be achieved using this guide, kindly written by the developer. - https://www.reddit.com/r/technitium/s/g3nRMvM1VA

It does sound like the scaling with kubernetes would be an implementation detail out with the scope of this. It would be down to your self to troubleshoot and implement a solution compatible with your infrastructure.

Being rude and demanding doesn't get you any further.

Give it a shot in a dev environment and feedback any issues.

[liamfleming26]

@Zorlin: Try it in a hyperscaled kuberneetes cluster and share your config if its working and I will stop moaning. sorry that I'm not working with 20 years old systems like others.

...and @liamfleming26 you can try it too. Or fork off :)

I'm a Lead Infrastructure Engineer and do some consultancy work too. Give me a shout and I can make some recommendations if you'd like.

[computerkoenig]

@liamfleming26 you're wrong. In the last hours I got a lot more maybe helpful replies than in the last 4 years ;) But keep trying!

And if you're able to help, it would be amazing! Thats all I ever wanted. I'll try to write a short summary the next days.

Poorly worded questions get poor answers.

Basic primary and secondary zone setups can be achieved using this guide, kindly written by the developer. - https://www.reddit.com/r/technitium/s/g3nRMvM1VA

It does sound like the scaling with kubernetes would be an implementation detail out with the scope of this. It would be down to your self to troubleshoot and implement a solution compatible with your infrastructure.

Being rude and demanding doesn't get you any further.

Give it a shot in a dev environment and feedback any issues.

[JL421]

Last response I have here until another update, or relevant discussion because I thought some people were taking the piss. I didn't realize how serious people can actually be.

@ShreyasZare I appreciate the response, I don't necessarily agree with your approach, but this is your baby, and it's fully in your right to do what you want with it.

...prioritization issues...

I fully get performance bugs and issues that just lock up the service are more important, I never said otherwise. That said, if you're manually managing "a couple of clusters" you should be like person number 1 wanting this finished, especially if it does take a lot of action on your part to maintain. I get that there are cool features out there, but waiting for an IETF draft to be finalized to implement a feature with many other solutions is definitely a choice.

single person development

Again, I respect the choice to limit who can contribute to the core codebase, that's why I also suggested other ways you can take load away from yourself. Most of them don't require advanced DNS knowledge to perform. Like just opening a Wiki and letting people contribute to it. Or issue templates that can get some of the "I can't setup this one feature that just hasn't been documented to the screenshot or full config file level" issues out of the way. Or breaking parts of goals you want to accomplish out in projects where someone might not need to have intricate codebase or DNS/DHCP knowledge to contribute. Think UI tweaks and usability improvements, packaging, etc.. Have someone help you setup build scripts so you don't have to spend time doing it manually. There's so much help you could get.

comparison to larger teams

Don't limit yourself like that. Sometimes in software it is better to reuse what works than build from the ground up. You could spend days writing a custom way to handle leap years, but why when there are libraries with 10s to 100s of contributors who already made one? Take what bits you can take that fit your needs, so long as there aren't any licensing issues you'd run into as a result.

All that said, I do appreciate the work you have put in so far, even if I don't fully agree with how you have and continue to work on this; I get it. Thanks for the updates.

[anonhostpi]

@ShreyasZare from one solo dev to another, take as much time as you need.

@JL421 You have to remember that solo devs do this as a hobby. Chances are @ShreyasZare has other things going on in his life, and even other things he may just want to spend his freetime on... (hanging out with friends, playing videogames, hiking, biking, exercising, vacays, etc).

There's no obligations for hobbyist solo devs to deliver features on a schedule.

[ShreyasZare]

@computerkoenig These comments you make tells more about yourself than anything else.

@ShreyasZare okay. most things I understand.

But

1. in my opinion is everyone a lier, who tells other people anything that will not happen. If there are complications or more importent things, you need to correct what you said or you're a lier. I always inform everyone I told that something will happen, which was not right/possible/...

Never said that this feature will never happen. Things getting delayed due to practical issues does not mean those were lies.

2. What the f... does this fucking DHCP feature do inside a DNS Server? Inside an really fast and load tollerant DNS deamon? Do you want to do everything, but nothing perfect?

If you do not want it then do not use it. These features were requested by many people and are being used widely.

3. If you would be able to communicate and talk to all the people wich are waiting and interrested, maybe there would be someone who is good enough (in your eyes) to just help you with this f***ing cluster feature. its really not as difficult, as a Mars mission. But maybe is this part of the reason why you are mostly alone working on this project.

Once you are open-minded and grown up enough to let us help, please let us know.

This is one of the reasons to avoid collaboration. Some people like to stall progress with unnecessary debates and strawman arguments. Any little time I have goes into this instead of implementing some features or fixing bugs.

Never did I say that it was difficult to implement. Its just a matter of time to implement it. The project has DNSSEC implemented which is multiple times more complex than clustering and took like 6 months to implement. Even building a recursive resolver is a lot complex than clustering.

Kind regards, Until a few hours ago, one of your longest admirers

[mtb-xt]

Once you are open-minded and grown up enough to let us help, please let us know.

This is one of the reasons to avoid collaboration. Some people like to stall progress with unnecessary debates and strawman arguments. Any little time I have goes into this instead of implementing some features or fixing bugs.

Another thing is that pressuring a single maintainer can be an attack, as demonstrated recently with xz archiver... @ShreyasZare ignore people and do your own thing.

[westdam]

Please , stop flaming and let the developer working on It.

Byez

Il sab 25 mag 2024, 12:17 Marat Bakeev @.***> ha scritto:

Once you are open-minded and grown up enough to let us help, please let us know.

This is one of the reasons to avoid collaboration. Some people like to stall progress with unnecessary debates and strawman arguments. Any little time I have goes into this instead of implementing some features or fixing bugs.

Another thing is that pressuring a single maintainer can be an attack, as demonstrated recently with xz archiver... @ShreyasZare https://github.com/ShreyasZare ignore people and do your own thing.

— Reply to this email directly, view it on GitHub https://github.com/TechnitiumSoftware/DnsServer/issues/134#issuecomment-2131187665, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP7YGHR7EPABENQ4DFVESM3ZEBQMHAVCNFSM4MOZHPLKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJTGEYTQNZWGY2Q . You are receiving this because you commented.Message ID: @.***>

[ShreyasZare]

Last response I have here until another update, or relevant discussion because I thought some people were taking the piss. I didn't realize how serious people can actually be.

@ShreyasZare I appreciate the response, I don't necessarily agree with your approach, but this is your baby, and it's fully in your right to do what you want with it.

It is quite normal to not agree with things and its ok. Here the approach is driven by a set of requirements which may not be the same for everyone.

...prioritization issues...

I fully get performance bugs and issues that just lock up the service are more important, I never said otherwise. That said, if you're manually managing "a couple of clusters" you should be like person number 1 wanting this finished, especially if it does take a lot of action on your part to maintain. I get that there are cool features out there, but waiting for an IETF draft to be finalized to implement a feature with many other solutions is definitely a choice.

The clusters I have manually setup do not require regular changes and all of the zones replicate to secondaries as configured. This works well here. Having clustering would make it easy to manage for sure.

single person development

Again, I respect the choice to limit who can contribute to the core codebase, that's why I also suggested other ways you can take load away from yourself. Most of them don't require advanced DNS knowledge to perform. Like just opening a Wiki and letting people contribute to it. Or issue templates that can get some of the "I can't setup this one feature that just hasn't been documented to the screenshot or full config file level" issues out of the way. Or breaking parts of goals you want to accomplish out in projects where someone might not need to have intricate codebase or DNS/DHCP knowledge to contribute. Think UI tweaks and usability improvements, packaging, etc.. Have someone help you setup build scripts so you don't have to spend time doing it manually. There's so much help you could get.

Ideally this sound good but practically, most of this do not work for such open source projects where long term contribution does not pay anything. I had someone who wanted to contribute by redesigning the web UI with many improvements. They did work on it for few months and eventually gave up. Even if they had completed it and added the UI upgrade to the project itself, the maintenance part suddenly becomes someone else's problem. I am not really much experienced with latest web/JS frameworks so this now becomes an issue that I have to spend time learning those libraries and frameworks to make changes to support new features.

For certain tangible features or tasks it works though. Like for example the windows installer part was added by @Cossey and it was really helpful.

There is almost no documentation (except some blog posts) due to time issue which is why I try to answer everyone on various forums like reddit, github, and over email. Having wiki will add another task to moderate it.

comparison to larger teams

Don't limit yourself like that. Sometimes in software it is better to reuse what works than build from the ground up. You could spend days writing a custom way to handle leap years, but why when there are libraries with 10s to 100s of contributors who already made one? Take what bits you can take that fit your needs, so long as there aren't any licensing issues you'd run into as a result.

A lot of DNS core stuff needs to have own library code in order to support adding features. Also, most of the common requirements get covered within the .NET library. I have avoided using 3rd party libaries for some tasks like Dashboard since it would have issues with performance and also cause dependency issues. Like for example. Sqlite library no longer works on some older Linux distros. Had the dashboard implementation relied on Sqlite then the DNS server would have become unusable for many deployments.

All that said, I do appreciate the work you have put in so far, even if I don't fully agree with how you have and continue to work on this; I get it. Thanks for the updates.

I appreciate your feedback. Thanks for the kind words.

[Cossey]

I tried designing an alternate UI too but new features were coming out for tDNS faster than I was able to do it 🙃. Not cool seeing some complaints for a feature that would typically be used by a large company or enterprise that would have the money to be able to pay Shreyas to prioritise it.

There was a good point raised though about documentation that the community should be able to assist with! I used bookdown on one of my repos; which takes markdown files; compiles a simple help website that builds using github workflows (though that project is designed to completely be built via GH and automation). Maybe something similar could be done for this project; that way it still uses pull requests and is moderated that way? https://github.com/Cossey/ScreenPixelRuler2 Look at the pushpr and release yml for the bookdown/bookdown-deploy/help jobs for an idea, builds from the help folder, gets deployed to https://cossey.github.io/ScreenPixelRuler2/help/master/

[Zorlin]

@Zorlin: Try it in a hyperscaled kuberneetes cluster and share your config if its working and I will stop moaning. sorry that I'm not working with 20 years old systems like others.

Something like this?

[anonhostpi]

Another thing is that pressuring a single maintainer can be an attack, as demonstrated recently with xz archiver... @ShreyasZare ignore people and do your own thing.

Good catch!

@ShreyasZare if any contributions are made from anyone on this issue, make sure they are NOT pressured, they are self-documenting, and don't contain unnecessary code!