oracle / vagrant-projects

Vagrant projects for Oracle products and other examples
Universal Permissive License v1.0
930 stars 473 forks source link

oracle-21c-vagrant project: vm seems to be working but have empty Oracle DB Home #475

Closed infiniteshi closed 1 year ago

infiniteshi commented 1 year ago

Describe the issue I have a oracle-21c-vagrant project that is named "oracle-21c-vagrant-3" locally and I tried to boot and login to it.

Below is what I ran, please note that I had multiple attempts and eventually my Oracle DB Home /opt/oracle/product/21c/dbhome_1/ directory is empty. me@myMac 21.3.0 % vagrant up Bringing machine 'oracle-21c-vagrant-3' up with 'virtualbox' provider... ==> oracle-21c-vagrant-3: Checking if box 'oraclelinux/8' version '8.7.411' is up to date... ==> oracle-21c-vagrant-3: Clearing any previously set forwarded ports... ==> oracle-21c-vagrant-3: Clearing any previously set network interfaces... ==> oracle-21c-vagrant-3: Preparing network interfaces based on configuration... oracle-21c-vagrant-3: Adapter 1: nat ==> oracle-21c-vagrant-3: Forwarding ports... oracle-21c-vagrant-3: 1521 (guest) => 1521 (host) (adapter 1) oracle-21c-vagrant-3: 5500 (guest) => 5500 (host) (adapter 1) oracle-21c-vagrant-3: 22 (guest) => 2222 (host) (adapter 1) ==> oracle-21c-vagrant-3: Running 'pre-boot' VM customizations... ==> oracle-21c-vagrant-3: Booting VM... ==> oracle-21c-vagrant-3: Waiting for machine to boot. This may take a few minutes... oracle-21c-vagrant-3: SSH address: 127.0.0.1:2222 oracle-21c-vagrant-3: SSH username: vagrant oracle-21c-vagrant-3: SSH auth method: private key ==> oracle-21c-vagrant-3: Machine booted and ready! [oracle-21c-vagrant-3] GuestAdditions 7.0.6 running --- OK. ==> oracle-21c-vagrant-3: Checking for guest additions in VM... ==> oracle-21c-vagrant-3: Setting hostname... ==> oracle-21c-vagrant-3: Mounting shared folders... oracle-21c-vagrant-3: /vagrant => /Users/ws2695/Documents/Backups/vm_backup/LocalDBNo3_OracleDatabase/21.3.0 ==> oracle-21c-vagrant-3: Machine already provisioned. Runvagrant provisionor use the--provision` ==> oracle-21c-vagrant-3: flag to force provisioning. Provisioners marked to run always will still run. me@myMac 21.3.0 % vagrant ssh

Welcome to Oracle Linux Server release 8.7 (GNU/Linux 5.15.0-5.76.5.1.el8uek.x86_64)

The Oracle Linux End-User License Agreement can be viewed here:

For additional packages, updates, documentation and community help, see:

Last login: Tue Mar 21 15:59:08 2023 from 10.0.2.2 [vagrant@oracle-21c-vagrant-3 ~]$ sudo su - oracle [oracle@oracle-21c-vagrant-3 ~]$ cd / [oracle@oracle-21c-vagrant-3 /]$ ls bin dev home lib64 mnt proc run srv tmp usr var boot etc lib media opt root sbin sys u01 vagrant [oracle@oracle-21c-vagrant-3 /]$ cd opt [oracle@oracle-21c-vagrant-3 opt]$ ls oracle oraInventory rh VBoxGuestAdditions-7.0.6 [oracle@oracle-21c-vagrant-3 opt]$ cd oracle/ [oracle@oracle-21c-vagrant-3 oracle]$ ls oracle product [oracle@oracle-21c-vagrant-3 oracle]$ cd product/ [oracle@oracle-21c-vagrant-3 product]$ ls 21c [oracle@oracle-21c-vagrant-3 product]$ cd 21c [oracle@oracle-21c-vagrant-3 21c]$ ls dbhome_1 [oracle@oracle-21c-vagrant-3 21c]$ cd dbhome_1/ [oracle@oracle-21c-vagrant-3 dbhome_1]$ ls [oracle@oracle-21c-vagrant-3 dbhome_1]$ exit logout [vagrant@oracle-21c-vagrant-3 ~]$ exit logout me@myMac 21.3.0 % vagrant provision ==> oracle-21c-vagrant-3: Running provisioner: shell... oracle-21c-vagrant-3: Running: /var/folders/ky/lvn5g3q53zl2mhjkcn0stw600000gq/T/vagrant-shell20230321-28527-1f0nkt.sh oracle-21c-vagrant-3: INSTALLER: Started up oracle-21c-vagrant-3: Last metadata expiration check: 1:26:49 ago on Tue 21 Mar 2023 02:59:04 PM -04. oracle-21c-vagrant-3: Dependencies resolved. oracle-21c-vagrant-3: Nothing to do. oracle-21c-vagrant-3: Complete! oracle-21c-vagrant-3: INSTALLER: System updated oracle-21c-vagrant-3: Last metadata expiration check: 1:26:52 ago on Tue 21 Mar 2023 02:59:04 PM -04. oracle-21c-vagrant-3: Dependencies resolved. oracle-21c-vagrant-3: ================================================================================ oracle-21c-vagrant-3: Package Arch Version Repository Size oracle-21c-vagrant-3: ================================================================================ oracle-21c-vagrant-3: Reinstalling: oracle-21c-vagrant-3: glibc-common x86_64 2.28-211.0.1.el8 ol8_baseos_latest 1.0 M oracle-21c-vagrant-3: oracle-21c-vagrant-3: Transaction Summary oracle-21c-vagrant-3: ================================================================================ oracle-21c-vagrant-3: oracle-21c-vagrant-3: Total download size: 1.0 M oracle-21c-vagrant-3: Installed size: 7.6 M oracle-21c-vagrant-3: Downloading Packages: oracle-21c-vagrant-3: glibc-common-2.28-211.0.1.el8.x86_64.rpm 1.3 MB/s | 1.0 MB 00:00 oracle-21c-vagrant-3: -------------------------------------------------------------------------------- oracle-21c-vagrant-3: Total 1.3 MB/s | 1.0 MB 00:00 oracle-21c-vagrant-3: Running transaction check oracle-21c-vagrant-3: Transaction check succeeded. oracle-21c-vagrant-3: Running transaction test oracle-21c-vagrant-3: Transaction test succeeded. oracle-21c-vagrant-3: Running transaction oracle-21c-vagrant-3: Preparing : 1/1 oracle-21c-vagrant-3: Reinstalling : glibc-common-2.28-211.0.1.el8.x86_64 1/2 oracle-21c-vagrant-3: Cleanup : glibc-common-2.28-211.0.1.el8.x86_64 2/2 oracle-21c-vagrant-3: Running scriptlet: glibc-common-2.28-211.0.1.el8.x86_64 2/2 oracle-21c-vagrant-3: Verifying : glibc-common-2.28-211.0.1.el8.x86_64 1/2 oracle-21c-vagrant-3: Verifying : glibc-common-2.28-211.0.1.el8.x86_64 2/2 oracle-21c-vagrant-3: oracle-21c-vagrant-3: Reinstalled: oracle-21c-vagrant-3: glibc-common-2.28-211.0.1.el8.x86_64 oracle-21c-vagrant-3: oracle-21c-vagrant-3: Complete! oracle-21c-vagrant-3: INSTALLER: Locale set oracle-21c-vagrant-3: INSTALLER: System time zone set to Etc/GMT+4 oracle-21c-vagrant-3: Last metadata expiration check: 1:26:56 ago on Tue 21 Mar 2023 02:59:04 PM -04. oracle-21c-vagrant-3: Package oracle-database-preinstall-21c-1.0-1.el8.x86_64 is already installed. oracle-21c-vagrant-3: Package openssl-1:1.1.1k-7.el8_6.x86_64 is already installed. oracle-21c-vagrant-3: Dependencies resolved. oracle-21c-vagrant-3: Nothing to do. oracle-21c-vagrant-3: Complete! oracle-21c-vagrant-3: INSTALLER: Oracle preinstall and openssl complete oracle-21c-vagrant-3: ln: failed to create symbolic link '/u01/app/oracle/oracle': File exists The SSH command responded with a non-zero exit status. Vagrant assumes that this means the command failed. The output for this command should be in the log above. Please read the output to determine what went wrong. me@myMac 21.3.0 % vagrant halt ==> oracle-21c-vagrant-3: Attempting graceful shutdown of VM... me@myMac 21.3.0 % vagrant up Bringing machine 'oracle-21c-vagrant-3' up with 'virtualbox' provider... ==> oracle-21c-vagrant-3: Checking if box 'oraclelinux/8' version '8.7.411' is up to date... ==> oracle-21c-vagrant-3: Clearing any previously set forwarded ports... ==> oracle-21c-vagrant-3: Clearing any previously set network interfaces... ==> oracle-21c-vagrant-3: Preparing network interfaces based on configuration... oracle-21c-vagrant-3: Adapter 1: nat ==> oracle-21c-vagrant-3: Forwarding ports... oracle-21c-vagrant-3: 1521 (guest) => 1521 (host) (adapter 1) oracle-21c-vagrant-3: 5500 (guest) => 5500 (host) (adapter 1) oracle-21c-vagrant-3: 22 (guest) => 2222 (host) (adapter 1) ==> oracle-21c-vagrant-3: Running 'pre-boot' VM customizations... ==> oracle-21c-vagrant-3: Booting VM... ==> oracle-21c-vagrant-3: Waiting for machine to boot. This may take a few minutes... oracle-21c-vagrant-3: SSH address: 127.0.0.1:2222 oracle-21c-vagrant-3: SSH username: vagrant oracle-21c-vagrant-3: SSH auth method: private key ==> oracle-21c-vagrant-3: Machine booted and ready! [oracle-21c-vagrant-3] GuestAdditions 7.0.6 running --- OK. ==> oracle-21c-vagrant-3: Checking for guest additions in VM... ==> oracle-21c-vagrant-3: Setting hostname... ==> oracle-21c-vagrant-3: Mounting shared folders... oracle-21c-vagrant-3: /vagrant => /Users/ws2695/Documents/Backups/vm_backup/LocalDBNo3_OracleDatabase/21.3.0 ==> oracle-21c-vagrant-3: Machine already provisioned. Run vagrant provision or use the --provision ==> oracle-21c-vagrant-3: flag to force provisioning. Provisioners marked to run always will still run. me@myMac 21.3.0 % vagrant ssh

Welcome to Oracle Linux Server release 8.7 (GNU/Linux 5.15.0-5.76.5.1.el8uek.x86_64)

The Oracle Linux End-User License Agreement can be viewed here:

For additional packages, updates, documentation and community help, see:

Last login: Tue Mar 21 16:25:04 2023 from 10.0.2.2 [vagrant@oracle-21c-vagrant-3 ~]$ sudo su - oracle [oracle@oracle-21c-vagrant-3 ~]$ sqlplus /nolog -bash: sqlplus: command not found [oracle@oracle-21c-vagrant-3 ~]$ cd /opt/oracle/product/21c/dbhome_1/ [oracle@oracle-21c-vagrant-3 dbhome_1]$

When I tried to connect to this DB on oracle-21c-vagrant-3 (it doesn't seem to make sense since there's no DB! but actually I had tried to connect to it before I found out about the empty Oracle DB Home) and had received the following error via my sql client:

image

I got the same error IO Error: Got minus one from a read call no matter the host is 127.0.0.1 or localhost or my actual fixed IP address like 192.168.x.y I even also tried to type in the host 10.0.2.2, inspired by the bold/highlighted line above Last login: Tue Mar 21 16:25:04 2023 from 10.0.2.2.

image

I also tried to type in the host 10.0.2.0 and 10.0.2.15, because: ` [vagrant@oracle-21c-vagrant-3 ~]$ ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255 inet6 fe80::a00:27ff:feef:c8d3 prefixlen 64 scopeid 0x20 ether 08:00:27:ef:c8:d3 txqueuelen 1000 (Ethernet) RX packets 11619 bytes 741409 (724.0 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 11369 bytes 724113 (707.1 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10 loop txqueuelen 1000 (Local Loopback) RX packets 6 bytes 416 (416.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 6 bytes 416 (416.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

[vagrant@oracle-21c-vagrant-3 ~]$ route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default _gateway 0.0.0.0 UG 100 0 0 eth0 10.0.2.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0 [vagrant@oracle-21c-vagrant-3 ~]$ ping _gateway PING _gateway (10.0.2.2) 56(84) bytes of data. 64 bytes from _gateway (10.0.2.2): icmp_seq=1 ttl=64 time=0.482 ms 64 bytes from _gateway (10.0.2.2): icmp_seq=2 ttl=64 time=0.449 ms 64 bytes from _gateway (10.0.2.2): icmp_seq=3 ttl=64 time=0.406 ms 64 bytes from _gateway (10.0.2.2): icmp_seq=4 ttl=64 time=0.490 ms ^C --- _gateway ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3053ms rtt min/avg/max/mdev = 0.406/0.456/0.490/0.042 ms ... `

Additional Information I found another bold-line highlighted above particularly noticeable - I feel like it's the roadblock. In another attempt, when I ran vagrant provision (to re-provision I guess?) I also got stuck at exactly the same line. For comparison, below is a happy-path console printout around about the same spot (with another vm "oracle-21c-vagrant-4): ... ... oracle-21c-vagrant-4: ... ... oracle-21c-vagrant-4: Complete! oracle-21c-vagrant-4: INSTALLER: Oracle preinstall and openssl complete oracle-21c-vagrant-4: INSTALLER: Oracle directories created oracle-21c-vagrant-4: INSTALLER: Environment variables set oracle-21c-vagrant-4: Archive: /vagrant/LINUX.X64_213000_db_home.zip oracle-21c-vagrant-4: creating: /opt/oracle/product/21c/dbhome_1/addnode/ oracle-21c-vagrant-4: inflating: /opt/oracle/product/21c/dbhome_1/addnode/addnode.sh oracle-21c-vagrant-4: inflating: /opt/oracle/product/21c/dbhome_1/addnode/addnode_oraparam.ini.sbs oracle-21c-vagrant-4: inflating: ... ...

Environment Vagrant 2.3.4 VirtualBox Guest Addition/inside the vm 7.0.6 (it was a mismatch before and that was even more problematic) VirtualBox on host 7.0.6r155176 Oracle Linux 8.7.411 Oracle DB 21.1.3 OS Ventura 13.2.1 Vagrantfile: currently exactly the same as in your project, no customization

Thank you very much! Any feedback would be helpful.

PaulNeumann commented 1 year ago

@infiniteshi

This VM may not be recoverable. But it's possible that running vagrant provision created an incorrect symbolic link, and the software and datafiles still exist. Running vagrant provision often does more harm than good when using the shell provisioner, as this project does. The shell script assumes that it will be run only once, which is why there was an error when it tried to create a symbolic link that already existed.

What output do you get when you run the following commands? (You can run them as either vagrant or oracle.)

infiniteshi commented 1 year ago

Hi @PaulNeumann , thanks a lot for the reply. I couldn't get to troubleshoot this issue again until today and to my surprise I can't reproduce my own issue after three weeks. There was consistent "context changes" to my laptop such as enterprise-compliant security patches and OS changes I would say.

It's very frustrating to admit that you might be right - This VM may not be recoverable. To be more exact, my old VM called "oracle-21c-vagrant-copy" that this "oracle-21c-vagrant-3" is trying to mimic or transplant from, is not recoverable. Basically what I did was trying to "copy and paste" only selected directories from "oracle-21c-vagrant-copy" to oracle-21c-vagrant-3". I tried to do this on different levels, including (1) the /vagrant-projects/ directory (2) the /vagrant-projects/OracleDatabase/21.3.0 directory (3) Instead of using a clean-slate LINUX.X64_213000_db_home.zip (about 3.1G in size) freshly downloaded from Oracle official site, used the zipped file that was compressed from my already application-data-populated LINUX.X64_213000_db_home (about 7.1G in size) of vm "oracle-21c-vagrant-copy"

None of these approaches were recommended by you or any other authors here, but trying them out silenced my curiosity "what if it works" and yielded a great variety of insight. And quite to my surprise, as of all my attempts today, none of the approaches above broke anything but always generated a clean-slate database for me each time - with newly generated new default passwords. And I could never re-access my application-data again.

To answer your question, with approach (3), below is what I got: [oracle@oracle-21c-vagrant-3 ~]$ ls /u01/app -lA

total 0
lrwxrwxrwx. 1 root root 11 Apr 11 15:37 oracle -> /opt/oracle

[oracle@oracle-21c-vagrant-3 ~]$ ls /opt/oracle/oracle ls: cannot access '/opt/oracle/oracle': No such file or directory [oracle@oracle-21c-vagrant-3 ~]$ ls /u01/app/oracle admin audit cfgtoollogs dbs diag homes oradata product

I feel pretty beaten and hopeless about fixing and would like to "temporarily" give up on it now (I say "temporarily" means I don't want to remove my broken vm with my data in it - that'd be completely giving up, but wait an infinite time to try again).