Closed Macleykun closed 3 years ago
pschapman
commented
3 years ago Hi Macleykun -
Yes. It is supported. If using internal templates in ZTP, just ensure that the end marker for your multi-line (e.g. banner) does not conflict with the end marker you are using for the "ztp set template" command.
PSC
Macleykun
commented
3 years ago Thanks for the fast reaction! I'll leave this open if i have some futher questions! I will close this once i finished my thesis/PoC!
Macleykun
commented
3 years ago So i've been testing a bit sence then, and i noticed if i comment out half the config, the first half seems to work better.
Is it possible that: it's only possible to publish an x amount of config? OR, and probably most likely. If a command is not supported/incorrect. Will it cause problems for futher commands perhaps?
pschapman
commented
3 years ago Hi -
Can you clarify your issue? Are you stating that you get a partial download to the target device and the download randomly stops?
Cisco devices will apply any received configuration after the download "stops" (regardless of completeness of the download).
Are you able to share the configuration that you are using?
PSC
Macleykun
commented
3 years ago So the issue is that not the whole configuration is applied, so for instance I noticed the motd messages and creation of a folder isn’t done, however configuration changes are made that occurs earlier/later.
I do notice it copy’s the whole folder (I don’t see in the step nor Cisco logs anything going wrong with the download). Is it possible to see full debug log of how freeztp/Cisco device gets configured by tftp?
The configuration I use is sadly not one I’m allowed to share sadly, I have a meeting tomorrow with someone who is familiar with Cisco more then me (CCNA, or higher).
I will send you an update tomorrow, want to get templating 100% down, before I move on to excel > CSV with freeztp.
pschapman
commented
3 years ago Hi - Understood.
You should have a bytes value match between the filesize column shown in ztp show download and the "bytes received" shown on the device console when it TFTP downloads the configuration. Should look something like this:
Loading ZTP-24716BC98F-confg from 10.0.0.12 (via Vlan1): !
[OK - 453 bytes]Depending on the platform you are working with failed commands may or may not show up on the device console, so detection of failed commands would be by comparison. I recommend that you create a sample merged configuration using ztp request merge-test <hostname/serial>, then manually paste the configuration into the device to verify that all the commands work.
Don't feel dismayed by this work. This is fairly standard troubleshooting for any type of automation. You don't realize how much you rely on the backspace key until you aren't allowed to use it.
PSC
Macleykun
commented
3 years ago Ok i did some testing and found the issue (i think) and got it working now with the company template!
So there's two parts to this: TACAS, if there's no TACAS server, applying this could be a reason why it wouldn't work/parse everything.
The generation/work for SSH keys might be an actual reason why the config didn't got trough all the way. Again i'm afraid i can't disclose the command's that are causing this, but i can say that a large value is used.
In the comming days i'll update this issue with the exact part(s) that caused in this case the config to not push fully. Hopefully with permission so i can show which command caused it exactly.
Thank you for all your kind words and helpful information!
pschapman
commented
3 years ago Hi -
Alter the order of your template. There is no rule that you have to input commands in the same order in which they display in a show running-config. Simply move your TACACS commands to the end of the script.
For SSH, that will only delay the application of the script, but should not prevent it. Once again, it can be moved to the end of your config script to ensure more timely application of other commands.
I have also found that vtp mode transparent may disrupt command flow. Putting it first or last seems to work well.
PSC
Macleykun
commented
3 years ago So i did some testing/verifying and found out about the following: on a 3850 sw the template first did a ssh rsa keypair-name keypairname followed by crypto key generate rsa usage-keys label keypairname with a very large modulus.
This process not only costed like 5 minutes or so, but due the order and latest version/command to generate itself. Might have caused that the key was never made or somewhere hidden.
i'm going to recheck that rsa command and probably use one that people normally used. Also gonna move that part to the bottom of the template :)
There's only one small question i have left, and that is that sometimes (like 9 out of 10) the serial ID isn't detect and default keystore is used.
Any reason why this could happen? Is there a config to maybe try a couple of times or indefinitely? Assume that trough a vm (virtualbox) a direct connection is to the sw on the mgmt interface.
pschapman
commented
3 years ago Hi -
SSH Keys Your process seems atypical to me. Below is a standard minimum for SSH (no AAA):
hostname myhostname
ip domain name mydomain.tld
crypto key generate rsa general modulus <360-4096>
username myuser privilege 15 secret mysecret
line vty 0 15
login localOptional, but highly recommended:
ip ssh version 2
ip ssh timeout 60
ip ssh server algorithm mac hmac-sha1
ip ssh server algorithm encryption aes128-ctr aes192-ctr aes256-ctr
line vty 0 15
transport input sshThe switch will always choose the SSH key which is attached to the FQDN for new sessions. It's possible that a 4k key could take several minutes to generate, but I find that 3850s have a pretty powerful processor. My experience has shown 2k keys taking about 10-15 seconds on the 3850 platform.
Validation of your keys is via show crypto key mypubkey rsa and checking for a key attached to the switch FQDN.
SNMP
Look at the ZTP log to understand what is happening. ztp show log Install SNMP tools on your ZTP server so that you can manually test (for Ubuntu 20 sudo apt install snmp will install tools but not service).
Install the initial template on a blank switch. Default initial template is:
hostname {{ autohostname }}
!
snmp-server community {{ community }} ROThe community is set by ztp set community <name>
Then query the switch from ztp using snmpwalk (command below shows ztp default community):
snmpwalk -v 2c -c secretcommunity <ip> 1.3.6.1.2.1.47.1.1.1.1.11
If OID ending 1000 is not the one that contains your serial, then configure an appropriate oid in ZTP to get the desired answer.
Here's from one of my systems:
root@ztp:~# snmpget -v 2c -c secretcommunity 192.0.2.4 1.3.6.1.2.1.47.1.1.1.1.11.1000
iso.3.6.1.2.1.47.1.1.1.1.11.1000 = STRING: "<redacted>"PSC
Macleykun
commented
3 years ago Dear PSC,
The ZTP i got the SSH working thanks to you! I only have to figure out the right rules (based on IP). I'm probably making some rookie mistake and need to take a closer look next week! SNMP wise, i understand that FreeZTP fetches the serialnumber, but i do wonder if delaying the keystore detection might actually help. As the issue happens sometimes. This is smth i will test after i got SSH with ACL working.
I'm excited to see how the CSV/Excel sheet should be made so that people can easily input the vars and make all the changes with CSV/Excel to make the right templates for all the switches :)
Also, is it possible to tell which device needs to grab which IOS bin file? I noticed i can only submit one general one at a time. These two questions are just something i'm wondering about as i haven't seen much info about this as of yet (i just finished all the documentation i had to do, i expect to ramp up work on FreeZTP alot!)
pschapman
commented
3 years ago Hi -
For SNMP, I have not found any significant issue with the timing. I did see that someone reported in issue #78 that there may be a query bug if you have a lot of OIDs specified (I would say more than 3, because most of my deployments have 3). Once again, check your ZTP logs to understand what is happening and when. If your issues persist, you should add onto the existing issue and provide log output. You can also attempt to monitor SNMP conversations using TCPDump from your ZTP console.
I recommend that you read the tips.md file, particularly sections 4 and 5 to answer your question about multi-platform upgrade.
If you are planning a larger deployment, you may want to read the practical guide I wrote to go with my JFIT-ZTP project. It includes a number of comments on using the external keystore which you may find interesting.
HTH PSC
Macleykun
commented
3 years ago Sorry for the lack of updates. Good news is that i submitted my thesis which is great xD So the i only have one ID specified, i believe it was merely a timing thing and it's fixed for now with the SNMP :).
I actually have/want to do it from an excel sheet to configure all the keystore's with their appropiate serial numbers of the switches :). this is something i'm finnaly going to work on tommorow!
Your tips look great! A little complex but i think it's good material for what will happen after my project/PoC is done enough!
The JFIT-ZTP guide also looks really great! I especially like that i can see this part: https://github.com/pschapman/JFIT-ZTP/blob/main/documentation/practical_stack_guide.md#static-keystore-data
Which shows how the excel sheet must be layed out for FreeZTP to work!
I do want to ask (which i believe will be my final question) how the Excel sheet should be formatted. I believe it has to be like this?
keystore_id | association | idarray | hostname | description | location Switch1 | SERIALNUMBER | Switch1 | SW1 | SW1 from customer | sea location etc etc
(ofcours the | are each in their respective colums)
Also again, thank you so much for your incredible helpful and quick responses! It truely keeps me inspired to keep going for perfection/full automation!
pschapman
commented
3 years ago Hi -
Working with ZTP I have found that the following combinations work:
As discussed in the page I linked, the column 'keystoreid' is the only mandatory column and needs to be first. If serials are placed in the external keystore then they use variables named 'idarrayx' where x is an integer value.
| keystore_id | myvar1 | myvarx |
|---|---|---|
| hostname1 | blah | blah |
HTH PSC
Macleykun
commented
3 years ago Aah i see, i think for the sake of simplicity and having endusers not deal with any if possibly no cli commands. I'll focus on placing all the vars, IDArray's and assocation in the external keystore!
Given your example, i do wonder if it's also possible to refer to refer to the template for each keystore, and associatie the serialnumber so that each switch get's it own config (like you don't want each switched to be named the same way haha).
How would your example be expanded exactly to achieve this? I have something down atm as followed:
| keystore_id | assocation | idarray | myvar1 | myvarx |
|---|---|---|---|---|
| hostname1 | my_template | SERIALNUM | blah | blah |
Sorry for my more beginner/vague questions, i'm familiar with templating/ansible a bit, but this is the first time for me really working on it alone and in a new solution :)
Macleykun
commented
3 years ago Ok so i've figured out how to format the CSV etc and set it up as external keystore. However, i tried the external keystore out, by removing the freeztp command's and for some reason the idarray wasn't working.
So i've exported the command's that i know work (tested a few times) and configured the csv file into the external keystore: ztp set external-keystore output.csv type csv ztp set external-keystore output.csv file 'output.csv
I then removed the command's that were responsible for detecting the serial number, the keystore and then the template. I was then left with the default config (still having the company template with the same name in place).
I rebooted the FreeZTP service and tested it. I noticed it didn't saw the matched Keystore which i found strange. The CSV is layed out as followed:
| keystore_id | association | idarray_1 | MANAGEMENT_INTERFACE |
|---|---|---|---|
| DEMO | OWN_TEMPLATE | SERIALNUM | GigabitEthernet 0/0 |
Is there perhaps another step that i'm forgetting so that the serialnumber can be picked up in the external keystore, making it able to find the right vars and template?
Doing external keystore test does show the output that if copied pasted directly into freeZTP. Will make the whole thing work again. doing a request merge with typing the serialnumber in manually, also works.
pschapman
commented
3 years ago Hi -
I don't see anything wrong with your format. If you do ztp request merge-test SERIALNUM you should receive the expected configuration. Note that you do need to do a ztp service restart every time you change the external keystore file.
Additionally, if you issue ztp request external-keystore-test <keystore name>, ZTP will show you how it would translate your file into actual ZTP commands. This will probably help you determine the root cause of any issues you are having.
As to your comments from yesterday... Yes, you can have a separate association per keystore, but you will find it administratively cumbersome. That is why Jinja2 templating is included in ZTP. It allows you to build a small number of flexible templates rather than a large number of single-use templates. (Examples: Prod-Access, Lab-Access, Prod-Core, Lab-Core, Edge-Router, WAN-Router)
PSC
Macleykun
commented
3 years ago I see, i do notice that the autocomplete doesn't tell the serialnumber, but if i type it it will regonise it and make the correct output. Is it possible that all the parts in the external keystore, should be removed from FreeZTP's commands? Like normally, i add the external keystore, and then clear the command's in FreeZTP inorder to check if it indeed uses the external keystore. Tommorow i can continue working on this, i'll use my own template instead of the company so i can share more exactly what i'm doing, the test requests do all work fine/perfect.
Thank you so much for your help again, it's incredible helpful!
pschapman
commented
3 years ago Hi -
Here is the base ZTP config I'm using now for a deployment. Hopefully this will answer your question.
#######################################################
ztp set suffix -confg
ztp set initialfilename network-confg
ztp set community secretcommunity
ztp set tftproot /etc/ztp/tftproot/
ztp set imagediscoveryfile freeztp_ios_upgrade
ztp set file-cache-timeout 10
# ## Change to default values here ##
ztp set snmpoid C2960_SERIAL 1.3.6.1.2.1.47.1.1.1.1.11.1001
ztp set snmpoid C3xx0_SERIAL 1.3.6.1.2.1.47.1.1.1.1.11.1000
ztp set snmpoid C9x00_SERIAL 1.3.6.1.2.1.47.1.1.1.1.11.2000
ztp set logging merged-config-to-mainlog enable
# ## Change to default values here ##
ztp set logging merged-config-to-custom-file disable
#
#
# ## Change to default values here ##
ztp set initial-template ^
hostname {{ autohostname }}
crypto key generate rsa general modulus 2048
username <myuser> privilege 15 secret <mysecret>
aaa new-model
aaa authentication login default local
aaa authorization exec default local
aaa authorization console
snmp-server community {{ community }} RO
end
^
#
#######################################################
#
ztp set dhcpd-option ztp-tftp-address code 150 type ip-address
#
[DHCP Scopes Truncated]
#
#######################################################
#
# ## Change to default values here ##
ztp set template DEFAULT_TEMPLATE ^
hostname {{ hostname }}
interface Vlan1
ip address dhcp
no shut
ip domain-name <mydomain>
username <myuser privilege 15 secret <mysecret>
aaa new-model
aaa authentication login default local
aaa authorization console
aaa authorization exec default local if-authenticated
crypto key generate rsa modulus 2048
ip ssh version 2
end
^
#
#######################################################
#
ztp set keystore DEFAULT_VALUES vl1_ip_address dhcp
ztp set keystore DEFAULT_VALUES hostname UNKNOWN_HOST
#
# ## Change to default values here ##
[ZTP sample keystore values deleted]
#
ztp set keystore GLOBAL ztp_env test
ztp set keystore GLOBAL c9k_ver 16.12.5b
ztp set keystore GLOBAL ztp_ip_addr <address>
ztp set keystore GLOBAL c9200_image cat9k_lite_iosxe.16.12.05b.SPA.bin
ztp set keystore GLOBAL c9k_image cat9k_iosxe.16.12.05b.SPA.bin
ztp set keystore GLOBAL c9klite_image cat9k_lite_iosxe.16.12.05b.SPA.bin
#
ztp set default-keystore DEFAULT_VALUES
ztp set global-keystore GLOBAL
ztp set default-template DEFAULT_TEMPLATE
ztp set imagefile NONE
ztp set image-supression 3600
ztp set delay-keystore 1000
#
# ## Change to default values here ##
[ZTP sample idarray values deleted]
#
[integrations truncated]
#
ztp set external-keystore EXT1 type csv
ztp set external-keystore EXT1 file '/home/root1/ztp/keystore.csv'
#
ztp set external-template EXT_BASELINE file '/home/root1/ztp/SwitchBaseline_v2a.txt'
#
#######################################################PSC
Macleykun
commented
3 years ago Heya!
So good news! I decided to try again for the night, and i noticed that in your config, you refer to your CSV file trough the full path. I only referd to the file as it was in the current directory (which it is).
I believe that thanks to this change and removing exactly the keystore values and idarray that i've gotten it to work!
I've also found a temporarly workaround which uses easy_install xlsx2csv to convert .xlsx to .csv which is formatted correctly by default for FreeZTP!
The last step i want/have to look into, is the upgrade of Cisco images. I'll test this out for myself, as i believe i have it configured the right way! However, i was curious if it's possible to change a template to download the appropiate image itself for each device. So that we don't have to change the image/bin each time a new device is provisioned that is different in model then the previous one!
pschapman
commented
3 years ago Hi -
Great. I'm going to mark this as closed. You can save Excel files to CSV by choosing the format in the Save As... dialog box.
PSC
Macleykun
commented
3 years ago I've tried that actually! But funny enough, it doesn't actually put the ( , ) between the words!
i was also curious if it's possible to make FreeZTP detect which image corresponds with which model. Now it seems that only one image/bin can be chosen, and it has to be changed for each new type of model :)
Hello, i've been testing FreeZTP for a while, and recently i got my own template working what was great! I did check on the switch itself, but noticed not everything was succesfully configured.
One of the things i know should always work, is the MOTD banner.
Wanted to know if this kind of stuff is supported, if so then i can look futher into my issue if it's a device specific problem, or related to FreeZTP.
The above example is changed from the template used, because the template is from a company i wish to not disclose :P
Looking forward to your answer!