Closed andrijapanicsb closed 1 year ago
it looks the route for mgmt cidr (10.0.32.0/20) is missing. the problem is, where can we find the gateway of mgmt cidr (e.g. 10.0.32.1) ? cannot find it in cmdline.
xenserver/kvm should not be impacted, as they use link local ip.
A critical export was moved from bootstrap.sh to init.sh
When the VPC is setup it asked in Line 64 for the hypvervisor https://github.com/apache/cloudstack/blob/296035d9a57caa62260a666da726ec45b6801c4b/systemvm/debian/opt/cloud/bin/setup/vpcrouter.sh#L64 The vpcrouter.sh gets called by the bootstrap.sh. In Version <= 4.16 it gets exported there and is available for vpcrouter.sh https://github.com/apache/cloudstack/blob/77df050160d6a5343aab97571bf891e51b4b1201/systemvm/debian/opt/cloud/bin/setup/bootstrap.sh#L215
The comit https://github.com/apache/cloudstack/commit/830f3061bcc33c0d3a22d9129e8bcf6a8ead37da removes the export in the bootstrap.sh and adds it to init.sh
A critical export was moved from bootstrap.sh to init.sh
When the VPC is setup it asked in Line 64 for the hypvervisor
The vpcrouter.sh gets called by the bootstrap.sh. In Version <= 4.16 it gets exported there and is available for vpcrouter.sh https://github.com/apache/cloudstack/blob/77df050160d6a5343aab97571bf891e51b4b1201/systemvm/debian/opt/cloud/bin/setup/bootstrap.sh#L215
The comit 830f306 removes the export in the bootstrap.sh and adds it to init.sh
@H-LK thanks for the findings.
prior to 4.17, the route is added in vpcrouter.sh which is called by cloud-early-config service. since 4.17, two steps are added before cloud-early-config
scp
command)The scp
in patch
step does not work, because the route is not added (cloud-early-config
is not executed yet).
The route should be added in init.sh
, instead of vpcrouter.sh
We tested current 4.18-vm-autoscaling branch 4.18.0.0-SNAPSHOT yesteday. Using kvm systemtemplate4.17.0 which modified cloudstack-release to 4.18.0,we found VR vm disappeared in ui. Using ssh connected the VR vm under agent host, it had console proxy java process, so acted as CPVM wrongly.
That is strange. Can you test it again ?
We pulled 4.18-vm-autoscaling branch on 2022-09-26, it worked well. Then we pulled on 2022-11-17, it failed cause VR cannot start. system vms virtual route vm view console, url is http://10.2.40.59/resource/noVNC/vnc.html?autoconnect=true&port=8080&token=F6IDkA3EnAfclsOgXqbytvUsEbKgC2jAWuP6ybG5HyqNgyR2fAwSFn9BbsCjgC28jQTR5O5v84f-T18N5Xmj-_otcsNv3I7sYLl9dN7HbM5LeVbxzaoG19e4DjEcVklBnN7_QclFRuV7NE-X9cp0YYLd9w6eWdQEjGKPvc4CxJaPqcZDn53gXVIUuj2-l7aTK7Wf4bnMnJ41oLNP39kqCaHiwJXskke7ciEcpIWzIJr3GXSGcfD67N5NOAXto0efGs2ZOKcpjkcw2B6cLj0_tepUNDdjehBq7TSHIjgWFu788xVA0Km2jY7HZbChqZ39icAfeNhL2SfVglKWg9BORTidPb51sYufJrbsv8V_MhfcMICUFZq-q5yPCtiLZTQpox_o65J5v-ed7r6Ft_rJbC3XbTwOv4P8nfNI_UZkCMfM41rLOyZRNBjhkFijXCHi7H9TX7v-93vsQq0cbK6Quw. Correctly it may be http://10.2.8.13:8080/client/console?cmd=access&vm=7545d234-0464-401b-a283-09c8d8b3f725 register iso timeout
we also tested main branch pulled on 2022-11-17, it had the same problem. How ever, Using virsh list it had not vrvm, it only had cpvm and ssvm. 4.18-vm-autoscaling had vrvm,but vrvm had cpvm java process.
we also tested main branch pulled on 2022-11-17, it had the same problem. How ever, Using virsh list it had not vrvm, it only had cpvm and ssvm. 4.18-vm-autoscaling had vrvm,but vrvm had cpvm java process.
@hooklee2000 did you test with a fresh installation or upgrade existing environment ?
We tested all with a fresh installation. By the way, using mysql 8, we must disable mysql's validate_password :
set global validate_password.policy=0;
set global validate_password.length=1;
otherwise cloudstack-setup-database printed error : We apologize for below error:
cloudstack-setup-database cloud:xxxxxx@127.0.0.1 --deploy-as=root:xxxxxx
***************************************************************
Encountering an error when executing mysql script
----------------------------------------------------------------------
table:
/usr/share/cloudstack-management/setup/create-database.sql
Error:
b"mysql: [Warning] Using a password on the command line interface can be insecure.\nERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)\n"
Sql parameters:
{'user': 'root', 'passwd': 'xxxxxx', 'port': 3306, 'host': '127.0.0.1'}
----------------------------------------------------------------------
***************************************************************
Please run:
cloudstack-setup-databases -h
for full help
We tested all with a fresh installation. By the way, using mysql 8, we must disable mysql's validate_password :
set global validate_password.policy=0; set global validate_password.length=1;
otherwise cloudstack-setup-database printed error : We apologize for below error:
cloudstack-setup-database cloud:xxxxxx@127.0.0.1 --deploy-as=root:xxxxxx *************************************************************** Encountering an error when executing mysql script ---------------------------------------------------------------------- table: /usr/share/cloudstack-management/setup/create-database.sql Error: b"mysql: [Warning] Using a password on the command line interface can be insecure.\nERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)\n" Sql parameters: {'user': 'root', 'passwd': 'xxxxxx', 'port': 3306, 'host': '127.0.0.1'} ---------------------------------------------------------------------- *************************************************************** Please run: cloudstack-setup-databases -h for full help
@hooklee2000 can you file a new github issue for this ?
ISSUE TYPE
COMPONENT NAME
CONFIGURATION
mgmt server is in Pod1/subnet1, hypervisor (and thus VR's control NIC) is in Pod2/subnet2 - thus there is a routing process between mgmt and VR's NIC IP
OS / ENVIRONMENT
Probably affects only VMware, but has not been tested with XS/KVM (should not be a problem, but needs checking)
SUMMARY
Only VPC VR affected (VR for shared or simpler Isolated network is NOT affected). There is a patching error of the VPC VR (this might be a problem or a consequence of missing routes) - either way, route from VR to the subnet of the mgmt server (subnet1) is missing/not configured, and thus VR can't talk to the mgmt server - VR configuration never completes. CMDLINE looks fine (all parameters look fine) but things are not applied correctly on the OS level.
STEPS TO REPRODUCE
EXPECTED RESULTS
ACTUAL RESULTS
CMD line of the VPC VR:
cloud.log
Routing table empty (there should be multiple routes (route to the mgmt cidr should be via 10.1.32.1 for 10.1.32.0 network - see CMDLINE output)