bingoohuang / blog

write blogs with issues
MIT License
177 stars 23 forks source link

linux commands #102

Open bingoohuang opened 5 years ago

bingoohuang commented 5 years ago

# https://superuser.com/questions/554319/display-each-sub-directory-size-in-a-list-format-using-one-line-command-in-bash
# 查看当前目录下的各个子目录大小

[root@tencent-beta18 logs]# du -h --max-depth=1
98M     ./bjca-api-statistics-tasks-app
173M    ./bjca-app-api-manager-server
512K    ./apollo
1.4M    ./bjca-app-msgcenter-server
23M     ./arthas
44K     ./bjca-app-config-cases-server
2.7M    ./bjca-bedrock-oss-app
184K    ./bjca-auth-code-server-app
84K     ./bjca-api-usercenter-server-app
44K     ./elastic-job
4.0K    ./arthas-cache
145M    ./bjca-app-usercenter-tasks
324K    ./bjca-api-statistics-server-app
63G     ./bjca-app-usercenter
1.8G    ./bjca-app-metrics-ump-server
13M     ./bjca-app-elastic-server
8.0K    ./simple-jar-demo-server
28K     ./bjca-auth-code-tasks-app
12M     ./bjca-app-xxx-server
65G     .
bingoohuang commented 5 years ago

ps aux

ps aux output meaning

➜  rig git:(master) ps aux | head -5
USER               PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
bingoobjca        1522  13.0  1.4  5568692 240992   ??  S    一09上午  22:33.13 /Applications/iTerm.app/Contents/MacOS/iTerm2
bingoobjca       10559   4.1  0.0  4371836   4960 s002  S    10:13上午   0:03.01 -zsh
bingoobjca        1361   3.8  1.3  6239448 218640   ??  S    一09上午  25:19.07 /Applications/QQ.app/Contents/MacOS/QQ
bingoobjca       78181   2.8  9.7 179434016 1629116   ??  S    四10上午  58:29.96 /Applications/GoLand.app/Contents/MacOS/goland 

See the ps man page for more info.

Difference between VSZ vs RSS memory usage

VSZ - Virtual Set Size

The Virtual Set Size is a memory size assigned to a process ( program ) during the initial execution. The Virtual Set Size memory is simply a number of how much memory a process has available for its execution.

RSS - Resident Set Size

As oppose to VSZ ( Virtual Set Size ), RSS is a memory currently used by a process. This is a actual number in kilobytes of how much RAM the current process is using.

简单的说VSZ代表的是虚拟记忆体的使用量,RSS代表实体记忆体的使用量。但是这里RSS的值,是包括shared memory的。Linux会把一些share library(lib*.so)载入记忆体中,而不同的process会共享部份的share library,也就是说这些share library只被Linux在载入一次,而process可以从重复引用。

ps aux | awk '$2 == pid { print $0 }

➜  rig git:(master) ps aux | awk '$2 == 1522 { print $3,$4,$6,$0 }'
7.9 1.4 240720 bingoobjca        1522   7.9  1.4  5567532 240720   ??  S    一09上午  22:51.44 /Applications/iTerm.app/Contents/MacOS/iTerm2

按RSS 倒排序 ps aux | sort -rn -k 6 |head -n 5

➜  rig git:(master) ps aux | sort -rn -k 6 |head -n 5
bingoobjca       78181   3.0  9.7 179445460 1631240   ??  S    四10上午  59:12.60 /Applications/GoLand.app/Contents/MacOS/goland
bingoobjca         510   0.0  2.7  6215516 458620   ??  S    一09上午  59:10.09 /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
bingoobjca       48414   1.1  2.1  4814908 359972   ??  S    三11上午  13:47.57 /Applications/VirtualBox.app/Contents/MacOS/VBoxHeadless --comment centos-kerb_default_1560232592051_15800 --startvm 82973c29-834c-497e-94f9-217e2eb7d49b --vrde config
bingoobjca        1522   5.4  1.7  5706412 285892   ??  S    一09上午  22:56.76 /Applications/iTerm.app/Contents/MacOS/iTerm2
bingoobjca       75281   0.0  1.7  5710188 285860   ??  S    四08上午   4:20.78 /Applications/WeChat.app/Contents/MacOS/WeChat
bingoohuang commented 5 years ago

The TLDR pages are a community effort to simplify the beloved man pages with practical examples. tldr tar

# bingoo @ 192 in ~/github/dog on git:master o [11:57:37]
$ tldr tar

tar

Archiving utility.
Often combined with a compression method, such as gzip or bzip.
More information: <https://www.gnu.org/software/tar>.

- Create an archive from files:
    tar -cf target.tar file1 file2 file3

- Create a gzipped archive:
    tar -czf target.tar.gz file1 file2 file3

- Extract an archive in a target directory:
    tar -xf source.tar -C directory

- Extract a gzipped archive in the current directory:
    tar -xzf source.tar.gz

- Extract a bzipped archive in the current directory:
    tar -xjf source.tar.bz2

- Create a compressed archive, using archive suffix to determine the compression program:
    tar -caf target.tar.xz file1 file2 file3

- List the contents of a tar file:
    tar -tvf source.tar

- Extract files matching a pattern:
    tar -xf source.tar --wildcards "*.html"
bingoohuang commented 5 years ago

list files order by size desc:

[betaoper@beta-hetong default]$ find . -type f  -exec du -h {} + | sort -r -h |head -n 10
16M ./2019-07-09/1148515944367390722Convert.PDF
12M ./2019-06-28/1144528088150097921.PDF
7.9M    ./2019-07-02/1146051722958139394Convert.PDF
7.2M    ./2019-06-28/1144476741342109698.PDF
5.2M    ./2019-07-01/1145605641686265857.PDF
4.6M    ./2019-07-02/1146051549095849986.PDF
4.6M    ./2019-07-02/1146050677066493953.PDF
4.1M    ./2019-07-02/1146057879357550594.DOCX
4.1M    ./2019-07-02/1146053558574313475.DOCX
3.5M    ./2019-07-04/1146604946106630145.DOCX
bingoohuang commented 5 years ago

指定用户名,进行svn操作: svn co URL --username xxx --password xxx

bingoohuang commented 4 years ago

查看linux中某个端口(port)是否被占用

  1. lsof lsof -i:3306
  2. netstat -anp|grep 3306
bingoohuang commented 4 years ago

查看局域网IP列表

方法1, nmap

Install nmap, sudo apt-get install nmap then nmap -sP 192.168.1.* or more commonly nmap -sn 192.168.1.0/24 will scan the entire .1 to .254 range

This does a simple ping scan in the entire subnet to see which hosts are online.

$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.7  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 52:54:00:ef:16:bd  txqueuelen 1000  (Ethernet)
        RX packets 8929283010  bytes 3287811980423 (2.9 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8071523788  bytes 3525672558442 (3.2 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1  (Local Loopback)
        RX packets 2761872043  bytes 1133502059424 (1.0 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2761872043  bytes 1133502059424 (1.0 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

$ nmap -sP 192.168.1.*

Starting Nmap 6.40 ( http://nmap.org ) at 2019-11-26 11:01 CST
Nmap scan report for hadoop3 (192.168.1.3)
Host is up (0.00030s latency).
Nmap scan report for 192.168.1.5
Host is up (0.00080s latency).
Nmap scan report for hadoop1 (192.168.1.7)
Host is up (0.000088s latency).
Nmap scan report for 192.168.1.8
Host is up (0.00044s latency).
Nmap scan report for 192.168.1.9
Host is up (0.00042s latency).
Nmap scan report for hadoop2 (192.168.1.11)
Host is up (0.00060s latency).
Nmap scan report for hadoop0 (192.168.1.14)
Host is up (0.00031s latency).
Nmap scan report for 192.168.1.16
Host is up (0.00055s latency).
Nmap done: 256 IP addresses (8 hosts up) scanned in 2.41 seconds

方法2 arp

Try following steps:

  1. Type ipconfig (or ifconfig on Linux) at command prompt. This will give you the IP address of your own machine. For example, your machine's IP address is 192.168.1.6. So your broadcast IP address is 192.168.1.255.
  2. Ping your broadcast IP address ping 192.168.1.255 (may require -b on Linux)
  3. Now type arp -a. You will get the list of all IP addresses on your segment.
$ arp -a
? (169.254.0.4) at fe:ee:ec:9b:c5:79 [ether] on eth0
? (169.254.0.74) at fe:ee:ec:9b:c5:79 [ether] on eth0
? (169.254.1.12) at fe:ee:ec:9b:c5:79 [ether] on eth0
? (172.18.0.6) at 02:42:ac:12:00:06 [ether] on br-8983f91a1c88
? (172.17.0.2) at 02:42:ac:11:00:02 [ether] on docker0
? (169.254.1.5) at fe:ee:ec:9b:c5:79 [ether] on eth0
? (169.254.0.23) at fe:ee:ec:9b:c5:79 [ether] on eth0
hadoop0 (192.168.1.14) at fe:ee:ec:9b:c5:79 [ether] on eth0
? (169.254.1.17) at fe:ee:ec:9b:c5:79 [ether] on eth0
? (172.18.0.8) at 02:42:ac:12:00:08 [ether] on br-8983f91a1c88
? (172.18.0.3) at <incomplete> on br-8983f91a1c88
? (169.254.0.15) at fe:ee:ec:9b:c5:79 [ether] on eth0
gateway (192.168.1.1) at fe:ee:ec:9b:c5:79 [ether] on eth0
? (172.18.0.10) at 02:42:ac:12:00:0a [ether] on br-8983f91a1c88
? (169.254.1.2) at fe:ee:ec:9b:c5:79 [ether] on eth0
? (169.254.0.2) at fe:ee:ec:9b:c5:79 [ether] on eth0
? (192.168.1.10) at fe:ee:ec:9b:c5:79 [ether] on eth0
hadoop3 (192.168.1.3) at fe:ee:ec:9b:c5:79 [ether] on eth0
? (169.254.0.55) at fe:ee:ec:9b:c5:79 [ether] on eth0
? (169.254.0.3) at fe:ee:ec:9b:c5:79 [ether] on eth0
hadoop2 (192.168.1.11) at fe:ee:ec:9b:c5:79 [ether] on eth0
? (192.168.1.118) at fe:ee:ec:9b:c5:79 [ether] on eth0
? (172.18.0.7) at <incomplete> on br-8983f91a1c88
bingoohuang commented 4 years ago

grep a file, but show several surrounding lines?

For BSD or GNU grep you can use -B num to set how many lines before the match and -A num for the number of lines after the match.

grep -B 3 -A 2 foo README.txt

If you want the same number of lines before and after you can use -C num.

grep -C 3 foo README.txt

This will show 3 lines before and 3 lines after.

-A and -B will work, as will -C n (for n lines of context), or just -n (for n lines of context... as long as n is 1 to 9).

bingoohuang commented 4 years ago

cat << EOF

To build on @Livven's answer, here are some useful combinations.

  1. variable substitution, leading tab retained, overwrite file, echo to stdout
tee /path/to/file <<EOF
${variable}
EOF
  1. no variable substitution, leading tab retained, overwrite file, echo to stdout
tee /path/to/file <<'EOF'
${variable}
EOF
  1. variable substitution, leading tab removed, overwrite file, echo to stdout
tee /path/to/file <<-EOF
    ${variable}
EOF
  1. variable substitution, leading tab retained, append to file, echo to stdout
tee -a /path/to/file <<EOF
${variable}
EOF
  1. variable substitution, leading tab retained, overwrite file, no echo to stdout
tee /path/to/file <<EOF >/dev/null
${variable}
EOF
  1. the above can be combined with sudo as well
sudo -u USER tee /path/to/file <<EOF
${variable}
EOF
bingoohuang commented 4 years ago
#!/bin/bash

# bccrack.sh

cd "/Applications/Beyond Compare.app/Contents/MacOS"

if [[ -f BCompare.real ]]; then
    exit 0
fi

mv BCompare BCompare.real

cat > BCompare <<'EOF'
#!/bin/bash

rm "/Users/$(whoami)/Library/Application Support/Beyond Compare/registry.dat"
"`dirname "$0"`"/BCompare.real $@
EOF

chmod +x BCompare
bingoohuang commented 4 years ago

如何使用Journalctl查看和操作Systemd日志

  1. journalctl -u service-name.service -u flag is short for --unit
  2. journalctl -u service-name.service -b to see only log messages for the current boot, -b is short for --boot
  3. journalctl -u service-name For things named .service, you can actually just use
  4. journalctl --since "2018-08-30 14:10:10" --until "2018-09-02 12:05:50"
  5. journalctl --since "-1h15min"
  6. journalctl --since "2015-01-10" --until "2015-01-11 03:00"
  7. journalctl --since yesterday
  8. journalctl -n 显示最近的10个条目
  9. journalctl -n 20 显示最近的20个条目
  10. journalctl _PID=8088 -o json
  11. journalctl --since 09:00 --until "1 hour ago"
  12. journalctl -f view a live log of new messages as they are collected

Thanks:

  1. journalctl man page
  2. Use journalctl to View Your System's Logs
  3. 如何使用Journalctl查看和操作Systemd日志
bingoohuang commented 4 years ago

Linux 网卡,防火墙,转发

Manipulating network interfaces, firewalling, and forwarding from Go.

Manipulating network interfaces, firewalling, and forwarding from Go.

The last few years of development on Linux have been exciting on the network front:

  1. NFTables - A high performance replacement for IPTables, NFTables provides a sophisticated (bytecode-based) rules engine, and the ability to make atomic rule changes (something that IPTables sorely lacked).
  2. Netlink - Netlink is a generic interface to features inside the kernel, but ends up being prevalent (and very convenient) for manipulating the network stack.
  3. Network Namespaces - Okay this isn't new, but network namespaces in-part enabled the containerization of server workloads, network sandboxing, and are just awesome in general.

Best of all, we can automate everything using Go.

The Basics: Configuring your interfaces

A pre-requisite to any funky networking is bringing the interface up. This is trivial from Go, thanks to the rtnetlink API, codified by vishvananda's fabulous netlink package.

// Get a structure describing the network interface.
localInterface, err := netlink.LinkByName("eth0")
if err != nil {
    // handle error
}

// Give the interface an address of 192.168.1.1, on a
// network with a 255.255.255.0 mask.
ipConfig := &netlink.Addr{IPNet: &net.IPNet{
  IP: net.ParseIP("192.168.1.1"),
  Mask: net.CIDRMask(24, 32)
}}
if err = netlink.AddrAdd(localInterface, ipConfig); err != nil {
    // handle error
}

// Setup the default route, so traffic that doesn't hit
// 192.168.1.(1-255) can be routed.
if err = netlink.RouteAdd(&netlink.Route{
    Scope:     netlink.SCOPE_UNIVERSE,
    LinkIndex: localInterface.Attrs().Index,
    Dst:       &net.IPNet{IP: gatewayIP, Mask: net.CIDRMask(32, 32)},
}); err != nil {
    // handle error
}

// Lastly, bring up the interface.
if err = netlink.LinkSetUp(localInterface); err != nil {
    // handle error
}

Under the hood, the following is occurring:

  1. A socket of family AF_NETLINK is opened.
  2. Using the NETLINK_ROUTE protocol and RTM_GETLINK / RTM_SETLINK messages, the package queries and sets link information.
  3. Lastly, the RTM_NEWROUTE message is used to set the default gateway.

If you want to read about the wire encoding for this & other netlink interfaces, have a look at Matt Layher's posts on the subject.

Using NFTables for packet filtering

NFTables, the successor to IPTables, is a highly-configurable rules engine for processing packets. It is configured via a netlink interface, much like the example above.

A primer on NFTables

nftables是新式的数据包过滤框架,从Linux内核3.13版本开始可用,旨在替代现用的iptables框架。Nftables HOWTO 中文翻译,更多nftables相比iptables到底改变了什么

First, some terminology:

  1. Tables - Tables in NFTables are simply a container. A table can be associated with a single address family (ie: IPv4, IPv6, inet. I use inet for most of my tables as it works on both IPv4 & IPv6 traffic). Rules are contained within chains, and chains are contained within a Table.
  2. Chains - An NFTables chain is a hook into Linux's packet processing pipeline. You attach your rules to a specific chain, which will cause the rules to be evaluated whenever a packet hits a certain point in Linux's pipeline. For instance, if you attach rules to a chain of type prerouting, the rules will be evaluated before Linux looks at routes & makes a routing decision. When you specify a chain, you need to provide a type. I don't know exactly what this means, but its pretty logical. If you want to filter packets, use the type filter. If you want to re-write routing information (IPTables calls this mangling) use type route. If you are using NAT features, use type NAT.
  3. Rules - Rules are where your filtering/forwarding/whatever logic go. We talk about building these in the next section.

NFTables packet pipeline (abridged version)

The triangles represent chains. Conceptually, it makes sense to think of them as hooks. image

Putting it all together

So, if you want to filter out certain IPv4 traffic from the internet from everything, you would put your rules in a prerouting chain with hook type filter, in a table with a name of your choosing. The table would have an address family of IPv4.

Hopefully this section gives you enough information to fill in the opaque-sounding fields. The rest is pretty straightforward.

Setting up NFTables from Go

I've talked for long enough, lets get into some code shall we? We use the nftables package by stapelberg, which is shaping up to be the canonical Go package for nftables (I'm biased thou, I'm sending PRs there too).

c := &nftables.Conn{}

// Lets create our table first.
myTable := c.AddTable(&nftables.Table{
    Family: nftables.TableFamilyIPv4,
    Name:   "myFilter",
})

// Now, we create a chain which we will add our filter
// rules to.
myChain := c.AddChain(&nftables.Chain{
    Name:     "myChain",
    Table:    myTable,
    Type:     nftables.ChainTypeFilter,
    Hooknum:  nftables.ChainHookInput,
    Priority: nftables.ChainPriorityFilter,
})

// Lets add a rule to the chain that loads the source
// address, and compares it to a hardcoded IP.
c.AddRule(&nftables.Rule{
  Table: myTable,
  Chain: myChain,
  Exprs: []expr.Any{
    // payload load 4b @ network header + 12 => reg 1
    &expr.Payload{
      DestRegister: 1,
      Base:         expr.PayloadBaseNetworkHeader,
      Offset:       12,
      Len:          4,
    },
    // cmp eq reg 1 0x0245a8c0
    &expr.Cmp{
      Op:       expr.CmpOpEq,
      Register: 1,
      Data:     net.ParseIP("192.168.1.53").To4(),
    },
    // [ immediate reg 0 drop ]
    &expr.Verdict{
      Kind: expr.VerdictDrop,
    },
  },
})

// Apply the above (commands are queued till a call to Flush())
if err := c.Flush(); err != nil {
  // handle error
}

Lets break this down.

  1. First, we construct a literal nftables.Conn. This connection contains all the state of our table.
  2. Next, we define and create our table. Aside from naming, the only field we have to choose here is the family. As this example highlights dropping traffic based on an IPv4 address, it makes sense for it to be in the IPv4 family.
  3. Creating the chain is where the fields get confusing. We create a chain in the table we previously created, with the following fields:

    • Filter type - Because we want to filter packets
    • Input chain (hook) - Because we want to drop packets before they enter the system (but not interfere with forwarded traffic)
    • Filter priority - I havent talked about priority yet, because you should seldom need to change it. Priorities other than Filter allows you to make your rules run before/after internal nftables operations. See the nftables wiki for more details.
  4. Next, we add a rule to drop packets where the source IP is 192.168.1.53. We break down the composition of rules in the next section.
  5. Lastly, we call Flush() to apply everything in one go (The table, chain, nor rules are not created till you call Flush). While this may seem like a confusing API design decision, batching NFTables changes into atomic transactions allows you to avoid strange behaviour, when packets are recieved between two NFTables changes. Congrats! You've just applied your first NFTables rule to the system!

NFTables Rules

I've avoided writing about these till now as they are a little confusing at first. That said, the rule engine is one of the most powerful features NFTables has on offer.

An NFTables rule is composed of a series of expressions, which are evaluated from beginning to end. If the expression checks some condition, and the condition evaluates to false, the remainder of the expressions are not evaluated (and hence any actions in later expressions are not applied).

Other NFTables rules perform some action. For instance expr.Drop instructs NFTables to drop the packet if it is evaluated (conversely, expr.Accept tells NFTables to accept the packet). There are also actions that increment counters, actions that mangle packets, actions that mark the connection or NAT it, and many more. Lets break down the example we saw before:

// payload load 4b @ network header + 12 => reg 1
&expr.Payload{
  DestRegister: 1,
  Base:         expr.PayloadBaseNetworkHeader,
  Offset:       12,
  Len:          4,
},

This expression loads 4 bytes (the size of an IPv4 address) from offset 12 of the packet's network header. If we look at the wikipedia page for an IPv4 packet, we can see that offset 12 is the source IP address. Neat!

There are 20, 32bit registers you can use to store state as your rule is evaluated. Register 0 specifies the verdict (ie: accept/drop packet) code, so don't use it unless you know what you are doing.

// cmp eq reg 1 0x0245a8c0
  &expr.Cmp{
    Op:       expr.CmpOpEq,
    Register: 1,
    Data:     net.ParseIP("192.168.1.53").To4(),
  },

The next blob of code compares the contents of register 1, permitting the rule to continue evaluation if the register contents match (CmpOpEq) the data (192.168.1.53). This has the effect of aborting rule evaluation if the source IP of the packet is not 192.168.1.53.

The last expression expr.Verdict simply drops the packet if the rule gets that far in evaluation (internally, this sets register 0). Since only packets with a source IP of 192.168.1.53 get this far, the expressions in this rule collectively add up to drop packets from 192.168.1.53.

Future posts

There's heaps more to cover but this post is getting long.

Linux versions

linux Longterm release kernels

Version Maintainer Released Projected EOL
5.4 Greg Kroah-Hartman & Sasha Levin 2019-11-24 Dec, 2021
4.19 Greg Kroah-Hartman & Sasha Levin 2018-10-22 Dec, 2020
4.14 Greg Kroah-Hartman & Sasha Levin 2017-11-12 Jan, 2024
4.9 Greg Kroah-Hartman & Sasha Levin 2016-12-11 Jan, 2023
4.4 Greg Kroah-Hartman & Sasha Levin 2016-01-10 Feb, 2022
3.16 Ben Hutchings 2014-08-03 Jun, 2020

LinuxVersions

CentOS versions

CentOS version Architectures RHEL base Kernel CentOS release date RHEL release date Delay (days)
7.5-1804 x86-64 7.5 3.10.0-862 10 May 2018 10 April 2018 31
7.6-1810 x86-64 7.6 3.10.0-957 3 December 2018 30 October 2018 34
7.7-1908 x86-64 7.7 3.10.0-1062 17 September 2019 6 August 2019 42
8.0-1905 x86-64, POWER8 (le), AArch64 8.0 4.18.0-80 24 September 2019 7 May 2019 140
bingoohuang commented 2 years ago

Print lines in file from the match line until end of file

  1. sed -n '/matched/,$p' file
  2. awk '/matched/,0' file

In Awk there is , -- the "between" operator(range patterns). Following some examples (default action = print)

you can freely mix regexps and line number predicates (or any expression) in them. For example:

  1. NR==1,/rex/ # all lines from the 1rst up to the one matching /rex/
  2. /rex/,0 # from the line matching /rex/ up to the end-of-file.