Open rovelynbancolo opened 6 years ago
Sentry:
Configure Sentry to recognize this account as an administrator Add your test user's primary group to the sentry.service.admin.group list in CM
rovy: data_engineers Added data_engineers to sentry.service.admin.group
Verify user privileges
172-31-0-136.ap-southeast-1.compute.internal@CLOUDERA.COM'
0: jdbc:hive2://ip-172-31-0-136.ap-southeast-> show tables;
+-----------+--+
| tab_name | +-----------+--+ +-----------+--+ No rows selected (0.36 seconds) 0: jdbc:hive2://ip-172-31-0-136.ap-southeast->
Create additional test users
On all host added this user and group details
kadmin -p admin/admin
addprinc george@CLOUDERA.COM addprinc ferdinand@CLOUDERA.COM addprinc rovy@CLOUDERA.COM addprinc root@CLOUDERA.COM addprinc hdfs@CLOUDERA.COM
Create test roles
!connect 'jdbc:hive2://ip-172-31-0-136.ap-southeast-1.compute.internal:10000/;principal=hive/ip-172-31-0-136.ap-southeast-1.compute.internal@CLOUDERA.COM' CREATE ROLE reads; CREATE ROLE writes;
Grant read privilege for all tables to reads
GRANT SELECT ON DATABASE default TO ROLE reads; GRANT ROLE reads TO GROUP selector;
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. SentryAccessDeniedException: root has no grant! (state=08S01,code=1)
!connect 'jdbc:hive2://ip-172-31-0-136.ap-southeast-1.compute.internal:10000/;principal=hive/ip-172-31-0-136.ap-southeast-1.compute.internal@CLOUDERA.COM'
GRANT SELECT ON DATABASE default TO ROLE reads; GRANT ROLE reads TO GROUP selector;
Grant read privilege for default.sample07 only to 'writes': Create a table under default. CREATE EXTERNAL TABLE IF NOT EXISTS default.sample_07( policyid VARCHAR(255) ,statecode VARCHAR(255) ,county VARCHAR(255) ,eq_site_limit VARCHAR(255) ,hu_site_limit VARCHAR(255) ,fl_site_limit VARCHAR(255) ,tiv_2011 VARCHAR(255) ,tiv_2012 VARCHAR(255) ,eq_site_deductible VARCHAR(255) ,hu_site_deductible VARCHAR(255) ,fl_site_deductible VARCHAR(255) ,fr_site_deductible VARCHAR(255) ,point_latitude VARCHAR(255) ,point_longitude VARCHAR(255) ,line VARCHAR(255) ,construction VARCHAR(255) ,point_granularity VARCHAR(255)) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE location '/user/hive/insurance_portfolio.csv';
load data inpath 'hdfs://ip-172-31-0-136.ap-southeast-1.compute.internal:8020/rovy/insurance_portfolio.csv' into table default.sample_07
!connect 'jdbc:hive2://ip-172-31-0-136.ap-southeast-1.compute.internal:10000/;principal=hive/ip-172-31-0-136.ap-southeast-1.compute.internal@CLOUDERA.COM'
REVOKE ALL ON DATABASE default FROM ROLE writes; GRANT SELECT ON default.sample_07 TO ROLE writes; GRANT ROLE writes TO GROUP inserters;
kinit as george, then login to beeline [rovy@ip-172-31-0-136 logs]$ kinit george@CLOUDERA.COM Password for george@CLOUDERA.COM: [rovy@ip-172-31-0-136 logs]$ klist Ticket cache: FILE:/tmp/krb5cc_1002 Default principal: george@CLOUDERA.COM
Valid starting Expires Service principal 05/17/2018 04:34:45 05/18/2018 04:34:45 krbtgt/CLOUDERA.COM@CLOUDERA.COM
!connect 'jdbc:hive2://ip-172-31-0-136.ap-southeast-1.compute.internal:10000/default;principal=hive/ip-172-31-0-136.ap-southeast-1.compute.internal@CLOUDERA.COM'
0: jdbc:hive2://ip-172-31-0-136.ap-southeast-> show tables; INFO : Compiling command(queryId=hive_20180517043939_fa110382-da27-474f-ac90-95960923d267): show tables INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from deserializer)], properties:null) INFO : Completed compiling command(queryId=hive_20180517043939_fa110382-da27-474f-ac90-95960923d267); Time taken: 0.049 seconds INFO : Executing command(queryId=hive_20180517043939_fa110382-da27-474f-ac90-95960923d267): show tables INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180517043939_fa110382-da27-474f-ac90-95960923d267); Time taken: 0.115 seconds INFO : OK +------------+--+ | tab_name | +------------+--+ | sample_07 | | sample_08 | +------------+--+ 2 rows selected (0.207 seconds) 0: jdbc:hive2://ip-172-31-0-136.ap-southeast->
[rovy@ip-172-31-0-136 logs]$ kinit ferdinand@CLOUDERA.COM Password for ferdinand@CLOUDERA.COM: [rovy@ip-172-31-0-136 logs]$ klist Ticket cache: FILE:/tmp/krb5cc_1002 Default principal: ferdinand@CLOUDERA.COM
Valid starting Expires Service principal 05/17/2018 04:41:17 05/18/2018 04:41:17 krbtgt/CLOUDERA.COM@CLOUDERA.COM [rovy@ip-172-31-0-136 logs]$ beeline
show tables;
+------------+--+ | tab_name | +------------+--+ | sample_07 | +------------+--+
yum -y install rng-tools ( RANDOM NUMBER GENERATOR )
systemctl enable rng-tools
yum -y install krb5-server krb5-libs pam_krb5
cd /var/lib/kerberos/krb5kdc
modify the kadm5.acl /admin@CLOUDERA.COM modify the kdc.conf [kdcdefaults] kdc_ports = 88 kdc_tcp_ports = 88
***modify the /etc/krb5.conf
[libdefaults] default_realm = CLOUDERA.COM dns_lookup_kdc = false dns_lookup_realm = false ticket_lifetime = 604800 renew_lifetime = 604800 forwardable = true default_tgs_enctypes = rc4-hmac default_tkt_enctypes = rc4-hmac permitted_enctypes = rc4-hmac udp_preference_limit = 1 kdc_timeout = 3000 [realms] CLOUDERA.COM = { kdc = ip-172-31-0-136.ap-southeast-1.compute.internal admin_server = ip-172-31-0-136.ap-southeast-1.compute.internal } [domain_realm]
****Creating an initial KDC database
systemctl enable krb5kdc kadmin
systemctl start krb5kdc kadmin
chkconfig krb5kdc on
chkconfig kadmin on
kadmin.local "addprinc -q admin/admin"
Enter password for this admin principal
Kerberos Ticket Lifetime ticket_lifetime : 7days
KDC Type :MIT KDC
Kerberos Security Realm default_realm CLOUDERA.COM
KDC Server Host kdc ip-172-31-0-136.ap-southeast-1.compute.internal
KDC Admin Server Host admin_server ip-172-31-0-136.ap-southeast-1.compute.internal
Manage krb5.conf through Cloudera Manager
HUE: Kerberos Ticket Renewer Health Suppress... Healthy Kerberos Ticket Renewer: 0. Concerning Kerberos Ticket Renewer: 0. Total Kerberos Ticket Renewer: 1. Percent healthy: 0.00%. Percent healthy or concerning: 0.00%. Critical threshold: 51.00%.
kadmin -p admin/admin
listprincs
modprinc -maxrenewlife 90days krbtgt/CLOUDERA.COM@CLOUDERA.COM modprinc -maxrenewlife 90days +allow_renewable hue/ip-172-31-3-221.ap-southeast-1.compute.internal@CLOUDERA.COM
CM -> HUE -> Instances -> Stop HUE Server -> Click on it -> Regenerate Keytab. Restart the services.
HUE YARN - Enable Kerberos Authentication for HTTP Web-Consoles HDFS Enable Kerberos Authentication for HTTP Web-Consoles
Sentry:
On all host added this user and group details
sudo groupadd selector
sudo groupadd inserters
sudo useradd -u 1100 -g selector george
sudo useradd -u 1200 -g inserters ferdinand
useradd rovy
groupadd data_engineers
usermod -g data_engineers rovy
kadmin -p admin/admin
addprinc george@CLOUDERA.COM
addprinc ferdinand@CLOUDERA.COM
addprinc rovy@CLOUDERA.COM
addprinc root@CLOUDERA.COM
addprinc hdfs@CLOUDERA.COM
[rovy@ip-172-31-0-136 ~]kinit rovy@CLOUDERA.COM [rovy@ip-172-31-0-136 ~] hdfs dfs -copyFromLocal *.csv /user/rovy/
Create a Hive database using HUE -> named the table insurance_profile. Access beeline.
beeline
!connect 'jdbc:hive2://ip-172-31-0-136.ap-southeast-1.compute.internal:10000/;principal=hive/ip-172-31-0-136.ap-southeast-1.compute.internal@CLOUDERA.COM'
load data inpath '/user/hive/warehouse/datalake.db/insurance_portfolio.csv' into table datalake.insurance_portfolio;
Enable Sentry
Create a Sentry Database
CREATE DATABASE sentry default character set utf8;
GRANT ALL ON sentry.* to 'sentry'@'%' identified by 'sentry_password';
flush privileges;
CM -> Add SEntry -> Enter DB Details
**Sentry Server Error -> it doesnt create table to the database*** Tried Creating the Database, remove sentry and readd it again. Query for candidates of org.apache.sentry.provider.db.service.model.MSentryVersion and subclasses resulted in no possible candidates Required table missing : "
SENTRY_VERSION
" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.autoCreateTables" org.datanucleus.store.rdbms.exceptions.MissingTableException: Required table missing : "SENTRY_VERSION
" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.autoCreateTables"--> Still Unable to Get Sentry Working.