Why Scraping the Website?

[elacy]

Why aren't you using the shell or the api

[marshallford]

Hey @elacy, thanks for your interest in my side project.

Why Scraping the Website?

Fair question. I partially started this project to tinker with golang and HTML scraping, as reckless as that might seem. A lot can be learned from diving into how the pfSense UI works and solving problems presented when dealing with scraping and strange form submission behavior.

shell

I was originally concerned about skipping over server validation that is circumvented when directly modifying the $config, but to your point its simplicity when combined with built-in JSON PHP functions shouldn't be overlooked. I am considering rewriting the various Get functions to perform at least the read operations using the shell. Thoughts?

api

I chose not to target the unofficial API because I didn't want the provider to have a third party dependency, nor was I excited about troubleshooting the intersection of pfSense, an extra pkg, and the Terraform plugin framework. That said, who knows -- maybe I'll change my mind as I roll out new features.

[elacy]

Fair question. I partially started this project to tinker with golang and HTML scraping, as reckless as that might seem. A lot can be learned from diving into how the pfSense UI works and solving problems presented when dealing with scraping and strange form submission behavior.

That's fair, hope you are having fun with it :D

I was originally concerned about skipping over server validation that is circumvented when directly modifying the $config, but to your point its simplicity when combined with built-in JSON PHP functions shouldn't be overlooked. I am considering rewriting the various Get functions to perform at least the read operations using the shell. Thoughts? I chose not to target the unofficial API because I didn't want the provider to have a third party dependency, nor was I excited about troubleshooting the intersection of pfSense, an extra pkg, and the Terraform plugin framework. That said, who knows -- maybe I'll change my mind as I roll out new features.

I suppose there are a few important things to consider:

• Any solution to this problem adds to the value proposition offered by pfsense
• It is not possible for Netgate to maintain the pfsense UI such that they do not break your code
• PFSense was not written with API calls in mind
• It is in the interest of netgate to make sure that anything that lives in the PFsense eco system is supported
• The PHP shell is supported by Netgate
• PHP itself can be updated breaking any of the scripts even if Netgate has the best intentions
• As you pointed out the PHP Shell has no validations
• There are issues with the API like it doesn't currently run on Netgate hardware without forcing the install to ignore the architecture
• Anyone who is building or working on the API will be incentivized to make sure that the API continues to work post upgrade, it's also open source

I'm leaning toward the API as the best mechanism at the moment.

[marshallford]

Hey @elacy, based on your comment around using the PHP shell I have refactored the various get functions to use the php variable $config instead of scraping HTML tables. Feel free to review or provide additional suggestions. Thanks!

https://github.com/marshallford/terraform-provider-pfsense/releases/tag/v0.4.0

Oh and one other question/point of interest, have you seen this now 3 year old blog post by Netgate engineering revealing a move to golang for the pfsense UI? I just stumbled upon it recently here -- do you think relying on the PHP shell still makes sense with this info in mind? At a minimum I will document that this provider is only tested with CE versions of pfSense.

[trevor-crowley]

appreciate the work here. Been playing with terraform to spin up k3s on proxmox on my home lab. like the idea of integrating local DNS changes with terraform on my CE pfSense as well. Proper, secure Networking is hard. Getting there.

[elacy]

Hey @elacy, based on your comment around using the PHP shell I have refactored the various get functions to use the php variable $config instead of scraping HTML tables. Feel free to review or provide additional suggestions. Thanks!

https://github.com/marshallford/terraform-provider-pfsense/releases/tag/v0.4.0

Nice, I fixed the problem where the API only works on x86, and have written a proof of concept that works against it https://github.com/elacy/terraform-provider-pfsense. I reckon that's probably going to be the best possible solution and they seemed game for pull requests so I'll probably invest there.

Oh and one other question/point of interest, have you seen this now 3 year old blog post by Netgate engineering revealing a move to golang for the pfsense UI? I just stumbled upon it recently here -- do you think relying on the PHP shell still makes sense with this info in mind? At a minimum I will document that this provider is only tested with CE versions of pfSense.

I have not but 3 years is quite a long time to rewrite UI etc, I imagine it got canned because of all the work they are putting into TNSR. I hope they reinvest back into a pfsense api but I'll not hold my breath.

appreciate the work here. Been playing with terraform to spin up k3s on proxmox on my home lab. like the idea of integrating local DNS changes with terraform on my CE pfSense as well. Proper, secure Networking is hard. Getting there.

Yeah same I'm hoping to build my entire homelab using only terraform, doesn't really work if you can't add the required firewall rules with just pfsense. Hard to make your network secure if it's hard to manage the configuration.

My plan is to define each device mac address, IP address, friendly name and what firewall access it will need. Then the terraform will generate the unifi and pfsense configuration that needs to exist to make that happen.

So that means I need to be able to update interfaces, VLANs, DHCP, firewall, free radius, switch configuration and probably a bunch of stuff I haven't thought of yet. I've noticed that accessing FreeRadius configuration is not something that's available by default, I suspect this will be challenging