EC2 Server Down - Githubissues

CSID-DGU / 2024-2-SCS4031-Happy-7

2024년 2학기 융합캡스톤디자인 행복해조 repo

MIT License

2 stars 4 forks source link

EC2 Server Down #20

Closed sholee-pt closed 1 week ago

sholee-pt commented 1 week ago

목적

예기치 못한 에러로 인해 강제 종료된 서버를 복구한다.

작업 상세 내용

연결 시도

서버 로그 확인
```
... 이상 생략
```

[[0;32m OK [0m] Started [0;1;39msystemd-journald.service[0m - Journal Service.

[[0;32m OK [0m] Mounted [0;1;39mdev-hugepages.mount[0m - Huge Pages File System.

[[0;32m OK [0m] Mounted [0;1;39mdev-mqueue.mount[0m - POSIX Message Queue File System.

[[0;32m OK [0m] Mounted [0;1;39msys-kernel-debug.mount[0m - Kernel Debug File System.

[[0;32m OK [0m] Mounted [0;1;39msys-kernel-tracing.mount[0m - Kernel Trace File System.

[[0;32m OK [0m] Finished [0;1;39mkeyboard-setup.service[0m - Set the console keyboard layout.

[[0;32m OK [0m] Finished [0;1;39mkmod-static-nodes.service[0mâ¦Create List of Static Device Nodes.

[[0;32m OK [0m] Finished [0;1;39mlvm2-monitor.service[0m - Moâ¦using dmeventd or progress polling.

[[0;32m OK [0m] Finished [0;1;39mmodprobe@configfs.service[0m - Load Kernel Module configfs.

[[0;32m OK [0m] Finished [0;1;39mmodprobe@dm_mod.service[0m - Load Kernel Module dm_mod.

[[0;32m OK [0m] Finished [0;1;39mmodprobe@drm.service[0m - Load Kernel Module drm.

[[0;32m OK [0m] Finished [0;1;39mmodprobe@efi_pstore.service[0m - Load Kernel Module efi_pstore.

[[0;32m OK [0m] Finished [0;1;39mmodprobe@fuse.service[0m - Load Kernel Module fuse.

[[0;32m OK [0m] Finished [0;1;39mmodprobe@loop.service[0m - Load Kernel Module loop.

[[0;32m OK [0m] Finished [0;1;39msystemd-fsck-root.service[0m - File System Check on Root Device.

     Mounting [0;1;39msys-fs-fuse-connections.mount[0m - FUSE Control File System...

     Mounting [0;1;39msys-kernel-config.mount[0m - Kernel Configuration File System...

[[0;32m OK [0m] Started [0;1;39msystemd-fsckd.service[0m - Fiâ¦stem Check Daemon to report status.

     Starting [0;1;39msystemd-remount-fs.servicâ¦[0munt Root and Kernel File Systems...

     Starting [0;1;39msystemd-tmpfiles-setup-deâ¦[0m Device Nodes in /dev gracefully...

[[0;32m OK [0m] Finished [0;1;39msystemd-modules-load.service[0m - Load Kernel Modules.

[[0;32m OK [0m] Mounted [0;1;39msys-fs-fuse-connections.mount[0m - FUSE Control File System.

[[0;32m OK [0m] Mounted [0;1;39msys-kernel-config.mount[0m - Kernel Configuration File System.

[[0;32m OK [0m] Finished [0;1;39msystemd-udev-trigger.service[0m - Coldplug All udev Devices.

     Starting [0;1;39msystemd-sysctl.service[0m - Apply Kernel Variables...

[[0;32m OK [0m] Finished [0;1;39msystemd-tmpfiles-setup-deâ¦[0mic Device Nodes in /dev gracefully.

[[0;32m OK [0m] Finished [0;1;39msystemd-sysctl.service[0m - Apply Kernel Variables.

[[0;32m OK [0m] Finished [0;1;39msystemd-remount-fs.servicâ¦[0mmount Root and Kernel File Systems.

     Starting [0;1;39mmultipathd.service[0m - Deviâ¦pper Multipath Device Controller...

     Starting [0;1;39msystemd-journal-flush.serâ¦[0msh Journal to Persistent Storage...

     Starting [0;1;39msystemd-random-seed.service[0m - Load/Save OS Random Seed...

     Starting [0;1;39msystemd-tmpfiles-setup-deâ¦[0meate Static Device Nodes in /dev...

[[0;32m OK [0m] Finished [0;1;39msystemd-random-seed.service[0m - Load/Save OS Random Seed.

[[0;32m OK [0m] Finished [0;1;39msystemd-tmpfiles-setup-deâ¦[0mCreate Static Device Nodes in /dev.

     Starting [0;1;39msystemd-udevd.service[0m - Râ¦ager for Device Events and Files...

[[0;32m OK [0m] Started [0;1;39mmultipathd.service[0m - Devicâ¦Mapper Multipath Device Controller.

[[0;32m OK [0m] Reached target [0;1;39mlocal-fs-pre.target[0mâ¦Preparation for Local File Systems.

     Mounting [0;1;39msnap-amazon\x2dssm\x2dageâ¦[0m amazon-ssm-agent, revision 7993...

     Mounting [0;1;39msnap-core18-2829.mount[0m - â¦t unit for core18, revision 2829...

     Mounting [0;1;39msnap-snapd-21759.mount[0m - â¦t unit for snapd, revision 21759...

[[0;32m OK [0m] Finished [0;1;39msystemd-journal-flush.serâ¦[0mlush Journal to Persistent Storage.

[[0;32m OK [0m] Mounted [0;1;39msnap-amazon\x2dssm\x2dagenâ¦[0mor amazon-ssm-agent, revision 7993.

[[0;32m OK [0m] Mounted [0;1;39msnap-core18-2829.mount[0m - Mount unit for core18, revision 2829.

[[0;32m OK [0m] Mounted [0;1;39msnap-snapd-21759.mount[0m - Mount unit for snapd, revision 21759.

[[0;32m OK [0m] Reached target [0;1;39msnapd.mounts.target[0m - Mounted snaps.

[[0;32m OK [0m] Started [0;1;39msystemd-udevd.service[0m - Ruâ¦anager for Device Events and Files.

[[0;32m OK [0m] Started [0;1;39msystemd-ask-password-consoâ¦[0mequests to Console Directory Watch.

[[0;32m OK [0m] Reached target [0;1;39mcryptsetup.target[0m - Local Encrypted Volumes.

[[0;32m OK [0m] Found device [0;1;39mdev-ttyS0.device[0m - /dev/ttyS0.

[[0;32m OK [0m] Found device [0;1;39mdev-disk-by\x2dlabel-â¦[0m - Amazon Elastic Block Store BOOT.

[[0;32m OK [0m] Found device [0;1;39mdev-disk-by\x2dlabel-â¦[0m - Amazon Elastic Block Store UEFI.

     Starting [0;1;39msystemd-fsck@dev-disk-by\â¦[0mCheck on /dev/disk/by-label/BOOT...

     Starting [0;1;39msystemd-fsck@dev-disk-by\â¦[0mCheck on /dev/disk/by-label/UEFI...

Checking in progress on 1 disk (0.0% complete) [[0;32m OK [0m] Finished [0;1;39msystemd-fsck@dev-disk-by\â¦[0mm Check on /dev/disk/by-label/BOOT.

[[0;32m OK [0m] Finished [0;1;39msystemd-fsck@dev-disk-by\â¦[0mm Check on /dev/disk/by-label/UEFI.

Checking in progress on 0 disks (100.0% complete) [[0;32m OK [0m] Listening on [0;1;39msystemd-rfkill.socket[0mâ¦ll Switch Status /dev/rfkill Watch.

     Mounting [0;1;39mboot.mount[0m - /boot...

[[0;32m OK [0m] Mounted [0;1;39mboot.mount[0m - /boot.

     Mounting [0;1;39mboot-efi.mount[0m - /boot/efi...

[[0;32m OK [0m] Mounted [0;1;39mboot-efi.mount[0m - /boot/efi.

[[0;32m OK [0m] Reached target [0;1;39mlocal-fs.target[0m - Local File Systems.

[[0;32m OK [0m] Listening on [0;1;39msystemd-sysext.socket[0mâ¦tension Image Management (Varlink).

     Starting [0;1;39mapparmor.service[0m - Load AppArmor profiles...

     Starting [0;1;39mconsole-setup.service[0m - Set console font and keymap...

     Starting [0;1;39mfinalrd.service[0m - Create â¦time dir for shutdown pivot root...

     Starting [0;1;39mplymouth-read-write.serviâ¦[0mymouth To Write Out Runtime Data...

     Starting [0;1;39msystemd-binfmt.service[0m - Set Up Additional Binary Formats...

     Starting [0;1;39msystemd-tmpfiles-setup.seâ¦[0me Volatile Files and Directories...

     Starting [0;1;39mufw.service[0m - Uncomplicated firewall...

[[0;32m OK [0m] Finished [0;1;39mconsole-setup.service[0m - Set console font and keymap.

[[0;32m OK [0m] Finished [0;1;39mfinalrd.service[0m - Create â¦untime dir for shutdown pivot root.

[[0;32m OK [0m] Finished [0;1;39mplymouth-read-write.serviâ¦[0mPlymouth To Write Out Runtime Data.

     Mounting [0;1;39mproc-sys-fs-binfmt_misc.mâ¦[0mcutable File Formats File System...

[[0;32m OK [0m] Finished [0;1;39mufw.service[0m - Uncomplicated firewall.

[[0;32m OK [0m] Mounted [0;1;39mproc-sys-fs-binfmt_misc.moâ¦[0mxecutable File Formats File System.

[[0;32m OK [0m] Finished [0;1;39msystemd-binfmt.service[0m - Set Up Additional Binary Formats.

[[0;32m OK [0m] Finished [0;1;39msystemd-tmpfiles-setup.seâ¦[0mate Volatile Files and Directories.

     Starting [0;1;39msystemd-resolved.service[0m - Network Name Resolution...

     Starting [0;1;39msystemd-update-utmp.serviâ¦[0mord System Boot/Shutdown in UTMP...

[[0;32m OK [0m] Finished [0;1;39msystemd-update-utmp.serviâ¦[0mecord System Boot/Shutdown in UTMP.

[[0;32m OK [0m] Started [0;1;39msystemd-resolved.service[0m - Network Name Resolution.

[[0;32m OK [0m] Reached target [0;1;39mnss-lookup.target[0m - Host and Network Name Lookups.

[[0;32m OK [0m] Finished [0;1;39mapparmor.service[0m - Load AppArmor profiles.

     Starting [0;1;39msnapd.apparmor.service[0m - â¦iles managed internally by snapd...

     Starting [0;1;39mcloud-init-local.service[0m â¦-init: Local Stage (pre-network)...

[[0;32m OK [0m] Finished [0;1;39msnapd.apparmor.service[0m - â¦ofiles managed internally by snapd.

[ 3.678557] cloud-init[457]: Cloud-init v. 24.3.1-0ubuntu0~24.04.2 running 'init-local' at Fri, 18 Oct 2024 12:39:56 +0000. Up 3.66 seconds. [[0;32m OK [0m] Finished [0;1;39mcloud-init-local.service[0m â¦ud-init: Local Stage (pre-network).

[[0;32m OK [0m] Reached target [0;1;39mnetwork-pre.target[0m - Preparation for Network.

     Starting [0;1;39msystemd-networkd.service[0m - Network Configuration...

[[0;32m OK [0m] Started [0;1;39msystemd-networkd.service[0m - Network Configuration.

[[0;32m OK [0m] Reached target [0;1;39mnetwork.target[0m - Network.

     Starting [0;1;39mcloud-init.service[0m - Cloud-init: Network Stage...

[ 5.069193] cloud-init[490]: Cloud-init v. 24.3.1-0ubuntu0~24.04.2 running 'init' at Fri, 18 Oct 2024 12:39:58 +0000. Up 5.05 seconds. [ 5.077910] cloud-init[490]: ci-info: +++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++ [ 5.081231] cloud-init[490]: ci-info: +--------+------+---------------------------+---------------+--------+-------------------+ [ 5.084427] cloud-init[490]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | [ 5.087625] cloud-init[490]: ci-info: +--------+------+---------------------------+---------------+--------+-------------------+ [ 5.090798] cloud-init[490]: ci-info: | ens5 | True | 172.31.10.234 | 255.255.240.0 | global | 02:92:01:23:bf:c5 | [ 5.093973] cloud-init[490]: ci-info: | ens5 | True | fe80::92:1ff:fe23:bfc5/64 | . | link | 02:92:01:23:bf:c5 | [ 5.097108] cloud-init[490]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . | [ 5.100298] cloud-init[490]: ci-info: | lo | True | ::1/128 | . | host | . | [ 5.103496] cloud-init[490]: ci-info: +--------+------+---------------------------+---------------+--------+-------------------+ [ 5.106702] cloud-init[490]: ci-info: +++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++ [ 5.108960] cloud-init[490]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+ [ 5.111260] cloud-init[490]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | [ 5.113537] cloud-init[490]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+ [ 5.115773] cloud-init[490]: ci-info: | 0 | 0.0.0.0 | 172.31.0.1 | 0.0.0.0 | ens5 | UG | [ 5.118038] cloud-init[490]: ci-info: | 1 | 172.31.0.0 | 0.0.0.0 | 255.255.240.0 | ens5 | U | [ 5.120304] cloud-init[490]: ci-info: | 2 | 172.31.0.1 | 0.0.0.0 | 255.255.255.255 | ens5 | UH | [ 5.122561] cloud-init[490]: ci-info: | 3 | 172.31.0.2 | 0.0.0.0 | 255.255.255.255 | ens5 | UH | [ 5.124815] cloud-init[490]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+ [ 5.127061] cloud-init[490]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++ [ 5.128932] cloud-init[490]: ci-info: +-------+-------------+---------+-----------+-------+ [ 5.130780] cloud-init[490]: ci-info: | Route | Destination | Gateway | Interface | Flags | [ 5.132638] cloud-init[490]: ci-info: +-------+-------------+---------+-----------+-------+ [ 5.134499] cloud-init[490]: ci-info: | 0 | fe80::/64 | :: | ens5 | U | [ 5.136379] cloud-init[490]: ci-info: | 2 | multicast | :: | ens5 | U | [ 5.138242] cloud-init[490]: ci-info: +-------+-------------+---------+-----------+-------+

... 이하 생략

Ubuntu 24.04.1 LTS ip-172-31-10-234 ttyS0


* 서버 로그 상으로는 별다른 문제가 확인되지 않아 AWS 콘솔로 접속 시도
    * 포트: ttyS0 (직렬 연결)

### EC2 강제 재부팅
* AWS 콘솔에서 여전히 인스턴스가 실행중임을 확인
* 강제 중지 후 재시작

## 참고 사항
[EC2 인스턴스가 중지 상태로 멈춘 이유](https://repost.aws/ko/knowledge-center/ec2-instance-stuck-stopping-state)

sholee-pt commented 1 week ago

서버 재부팅 후(약 10분 소요), crontab 및 nohup을 통해 자동화 시스템 복구 완료