search

alibaba / euler

A distributed graph deep learning framework.
Apache License 2.0
2.9k stars 558 forks source link

是否可以导出feature的emebedding？ #159

Closed wentixiaogege closed 5 years ago

wentixiaogege commented 5 years ago

您好，问题如title，之所以需要feature的embedding是希望根据这做一些cold start的相关研究；不知道是否目前支持？

alinamimi commented 5 years ago

feature的embedding应该是在tf的model里面

wentixiaogege commented 5 years ago

是说，可以读取model自己的一个embedding信息？名字啥的能够找到的对么？有木有例子呀

formath commented 5 years ago

euler只是提供图存储和一些图上的操作op，这种需求自己实现即可，和使用tf开发普通模型一样

wentixiaogege commented 5 years ago

我感觉不是这么简单吧 [('beta1_power', []), ('beta2_power', []), ('global_step', []), ('graphsage_1/sageencoder_1/meanaggregator_1/dense_1/kernel', [30, 128]), ('graphsage_1/sageencoder_1/meanaggregator_1/dense_1/kernel/Adam', [30, 128]), ('graphsage_1/sageencoder_1/meanaggregator_1/dense_1/kernel/Adam_1', [30, 128]), ('graphsage_1/sageencoder_1/meanaggregator_1/dense_2/kernel', [30, 128]), ('graphsage_1/sageencoder_1/meanaggregator_1/dense_2/kernel/Adam', [30, 128]), ('graphsage_1/sageencoder_1/meanaggregator_1/dense_2/kernel/Adam_1', [30, 128]), ('graphsage_1/sageencoder_1/meanaggregator_2/dense_3/kernel', [256, 128]), ('graphsage_1/sageencoder_1/meanaggregator_2/dense_3/kernel/Adam', [256, 128]), ('graphsage_1/sageencoder_1/meanaggregator_2/dense_3/kernel/Adam_1', [256, 128]), ('graphsage_1/sageencoder_1/meanaggregator_2/dense_4/kernel', [256, 128]), ('graphsage_1/sageencoder_1/meanaggregator_2/dense_4/kernel/Adam', [256, 128]), ('graphsage_1/sageencoder_1/meanaggregator_2/dense_4/kernel/Adam_1', [256, 128]), ('graphsage_1/sageencoder_2/meanaggregator_3/dense_5/kernel', [30, 128]), ('graphsage_1/sageencoder_2/meanaggregator_3/dense_5/kernel/Adam', [30, 128]), ('graphsage_1/sageencoder_2/meanaggregator_3/dense_5/kernel/Adam_1', [30, 128]), ('graphsage_1/sageencoder_2/meanaggregator_3/dense_6/kernel', [30, 128]), ('graphsage_1/sageencoder_2/meanaggregator_3/dense_6/kernel/Adam', [30, 128]), ('graphsage_1/sageencoder_2/meanaggregator_3/dense_6/kernel/Adam_1', [30, 128]), ('graphsage_1/sageencoder_2/meanaggregator_4/dense_7/kernel', [256, 128]), ('graphsage_1/sageencoder_2/meanaggregator_4/dense_7/kernel/Adam', [256, 128]), ('graphsage_1/sageencoder_2/meanaggregator_4/dense_7/kernel/Adam_1', [256, 128]), ('graphsage_1/sageencoder_2/meanaggregator_4/dense_8/kernel', [256, 128]), ('graphsage_1/sageencoder_2/meanaggregator_4/dense_8/kernel/Adam', [256, 128]), ('graphsage_1/sageencoder_2/meanaggregator_4/dense_8/kernel/Adam_1', [256, 128])]

通过保存的ckpt里根本找不到feature对应的一个embedding情况，然后我看源码上，这两块好像是根本没有对uint64feature这种sparce的做一个embedding降低维度的操作么？我之前以为可以通过model然后tf.nn.embedding_lookup 之类的来走呢

void LocalGraph::GetNodeUint64Feature( const std::vector& node_ids, const std::vector& fids, std::function<void(const UInt64FeatureVec&)> callback) const { std::vector values; std::vector valuenums; UInt64FeatureVec features; engine->GetNodeUint64Feature(node_ids, fids, &value_nums, &values); BuildFeatureVec(values, value_nums, node_ids.size(), fids.size(), &features); callback(features); }

template <typename ValueType, typename FeatureVec> void BuildFeatureVec(const std::vector& values, const std::vector& value_nums, int ids_size, int fids_size, FeatureVec results) { auto data = values.data(); results->resize(ids_size); for (int i = 0; i < ids_size; ++i) { auto& result = results->at(i); result.resize(fids_size); for (int j = 0; j < fids_size; ++j) { auto& item = result[j]; item.resize(value_nums[i fids_size + j]); memcpy(&item[0], data, sizeof(item[0]) * item.size()); data += item.size(); } } }

zakheav commented 5 years ago

euler里存储的sparse feature是sparse feature的id，没有存储embedding。

wentixiaogege commented 5 years ago

最初传入的就是id吧，模型中间不应该有一个层，代表了embedding类似的东东么？

wentixiaogege commented 5 years ago

可以加个微信聊么，wentixiaogege 哈哈，大佬请教啊

zakheav commented 5 years ago

你可以这么理解，最初传入的叫做feature name，每路feature name里面的feature值我们叫做feature的id，可以用这个id去查embedding。embedding没有存储在euler里面，euler只存储了feature的id值。

alinamimi commented 5 years ago

在这里设置的embedding https://github.com/alibaba/euler/blob/e5387e1a4e8345ffd6f7e39fc4052a7abf6f7b56/tf_euler/python/encoders.py#L112

wentixiaogege commented 5 years ago

比如euler训练好了一个grapsage模型，我是不是可以根据这个feature id 得到这个feature在训练好的模型里面的embedding，还是说euler训练好的model本身也没有这个呀

wentixiaogege commented 5 years ago

alinamimi 大佬，所以说是有的，但是我为啥tf.train.list_variables(ckpt) 这样子找不到呢

wentixiaogege commented 5 years ago

我用的是graphsage unsupervised -》 调用了encoders.SageEncoder -》ShallowEncoder(layers.Layer)这样应该是可以获取到sparcefeature对应的embedding信息才对

alinamimi commented 5 years ago

需要设置sparse_feature_idx和use_hash_embedding，默认是不用的

wentixiaogege commented 5 years ago

python -m tf_euler --data_dir data --max_id 17204 --feature_idx 0 --feature_dim 30 --sparse_feature_idx 0 --use_hash_embedding True --model graphsage --mode train 这样么？

alinamimi commented 5 years ago

https://github.com/alibaba/euler/blob/e5387e1a4e8345ffd6f7e39fc4052a7abf6f7b56/tf_euler/python/run_loop.py#L283 这个是个例子，需要使用sparse feature，需要改一下graphsage model的参数调用

wentixiaogege commented 5 years ago

哦哦，我知道了，那怎么读取这个embedding的啊，我先跑出来这个看看；好像tf.train.list_variables(ckpt) 这种命令还是找不到咱们的embedding信息

alinamimi commented 5 years ago

embedding 信息就写在tf的model里面了

wentixiaogege commented 5 years ago

elif flags_obj.model == 'graphsage': model = models.GraphSage( node_type=flags_obj.train_node_type, edge_type=flags_obj.train_edge_type, max_id=flags_obj.max_id, xent_loss=flags_obj.xent_loss, num_negs=flags_obj.num_negs, metapath=metapath, fanouts=fanouts, dim=flags_obj.dim, aggregator=flags_obj.aggregator, concat=flags_obj.concat, feature_idx=flags_obj.feature_idx, feature_dim=flags_obj.feature_dim, sparse_feature_idx=[0,1], use_hash_embedding=True) 大佬是这样的吧？那model里面要怎么找啊，我那个方式不行么？

wentixiaogege commented 5 years ago

for node in G.nodes():

print('dealing with node',node)

buf = {}
node = int(node)
buf["node_id"] = node
buf["node_type"] = node_type_id(G.node[node])
out_vec[node_type_id(G.node[node])].write(str([node]) + '\n')
buf["node_weight"] = len(G[node]) if with_weight else 1
buf["neighbor"] = {}
for i in range(0, meta["edge_type_num"]):
    buf["neighbor"][str(i)] = {}
for n in G[node]:
    n=int(n)
    buf["neighbor"][str(edge_type_id(G[node][n]))][str(n)] = 1
buf["uint64_feature"]={}
buf["uint64_feature"][0] = list(feature[feature.id == node].price_cat_lable.values)
buf["uint64_feature"][1] = list(feature[feature.id == node].cate3_label.values)
buf["float_feature"] = {}
buf["float_feature"][0] = feature[feature.id == node].name_feature.values[0]
buf["binary_feature"] = {}
buf["edge"] = []
for tar in G[node]:
    tar = int(tar)
    ebuf = {}
    ebuf["src_id"] = node
    ebuf["dst_id"] = tar
    ebuf["edge_type"] = edge_type_id(G[node][tar])
    ebuf["weight"] = G[node][tar]['weight']
    ebuf["uint64_feature"] = {}
    ebuf["float_feature"] = {}
    ebuf["binary_feature"] = {}
    buf["edge"].append(ebuf)
out.write(json.dumps(buf) + '\n')

这是我构建图的脚本

wentixiaogege commented 5 years ago

ValueError: sparse_feature_idx must be the same length assparse_feature_max_id. 这是说，我有两个sparse feature的话要传入【第一个maxid，第二个maxid】么？

alinamimi commented 5 years ago

sparse_feature_max_id 需要指定 [第一个maxid，第二个maxid]

wentixiaogege commented 5 years ago

model = models.GraphSage( node_type=flags_obj.train_node_type, edge_type=flags_obj.train_edge_type, max_id=flags_obj.max_id, xent_loss=flags_obj.xent_loss, num_negs=flags_obj.num_negs, metapath=metapath, fanouts=fanouts, dim=flags_obj.dim, aggregator=flags_obj.aggregator, concat=flags_obj.concat, feature_idx=flags_obj.feature_idx, feature_dim=flags_obj.feature_dim, sparse_feature_max_id=[29,321], use_hash_embedding=True) 我试试这个

wentixiaogege commented 5 years ago

这样的话确实成功了：

flags_obj.train_node_type 0 flags_obj.train_edge_type [0] flags_obj.max_id 17204 flags_obj.xent_loss True flags_obj.num_negs 5 flags_obj.dim 256 flags_obj.aggregator mean flags_obj.concat True flags_obj.feature_idx 0 flags_obj.feature_dim 30 flags_obj.fanouts [10, 10] WARNING: Logging before InitGoogleLogging() is written to STDERR I1015 17:05:54.909902 4052 graph_builder.cc:81] Load Done: hlg/hlg_data.dat I1015 17:05:54.910136 4044 graph_builder.cc:127] Each Thread Load Finish! Node Count:7534 Edge Count:11261490 I1015 17:05:54.910181 4044 graph_builder.cc:135] Graph Loading Finish! I1015 17:06:01.876092 4044 graph_builder.cc:147] Graph Load Finish! Node Count:7534 Edge Count:11261490 I1015 17:06:06.816910 4044 graph_builder.cc:159] Done: build all sampler I1015 17:06:06.816958 4044 graph_builder.cc:162] Graph build finish WARNING:tensorflow:use_feature is deprecated and would not have any effect. WARNING:tensorflow:use_feature is deprecated and would not have any effect. WARNING:tensorflow:From /root/anaconda2/lib/python2.7/site-packages/tf_euler/python/base_layers.py:78: init (from tensorflow.python.ops.init_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.initializers.variance_scaling instead with distribution=uniform to get equivalent behavior. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. 2019-10-15 17:06:07.912727: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA INFO:tensorflow:Restoring parameters from ckpt/model.ckpt-3302 INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Saving checkpoints for 3302 into ckpt/model.ckpt. INFO:tensorflow:loss = 1348.1478, mrr = 0.43359375, step = 3303 INFO:tensorflow:Saving checkpoints for 3303 into ckpt/model.ckpt

读取embedding 要怎么读取呢； 我看咱们有一个run_save_embedding的方法，我要自己写一个这样的么？还是可以tf 直接找到？

alinamimi commented 5 years ago

获取node的 embedding 参考https://github.com/alibaba/euler/wiki/%E5%BF%AB%E9%80%9F%E5%BC%80%E5%A7%8B#4-embedding%E8%BE%93%E5%87%BA

feature 本事的embedding，是在tf model里面

wentixiaogege commented 5 years ago

大佬，这个embedding应该也在tf model里面，为何你们不直接从tf model里面读取呢？

加载checkpoint

sess = tf.Session() ckpt = tf.train.latest_checkpoint("./ckpt") saver = tf.train.import_meta_graph(ckpt + '.meta', clear_devices=True) saver.restore(sess, ckpt)

找到所有的变量

tf.train.list_variables(ckpt) [('beta1_power', []), ('beta2_power', []), ('global_step', []), ('graphsage_1/sageencoder_1/meanaggregator_1/dense_1/kernel', [30, 128]), ('graphsage_1/sageencoder_1/meanaggregator_1/dense_1/kernel/Adam', [30, 128]), ('graphsage_1/sageencoder_1/meanaggregator_1/dense_1/kernel/Adam_1', [30, 128]), ('graphsage_1/sageencoder_1/meanaggregator_1/dense_2/kernel', [30, 128]), ('graphsage_1/sageencoder_1/meanaggregator_1/dense_2/kernel/Adam', [30, 128]), ('graphsage_1/sageencoder_1/meanaggregator_1/dense_2/kernel/Adam_1', [30, 128]), ('graphsage_1/sageencoder_1/meanaggregator_2/dense_3/kernel', [256, 128]), ('graphsage_1/sageencoder_1/meanaggregator_2/dense_3/kernel/Adam', [256, 128]), ('graphsage_1/sageencoder_1/meanaggregator_2/dense_3/kernel/Adam_1', [256, 128]), ('graphsage_1/sageencoder_1/meanaggregator_2/dense_4/kernel', [256, 128]), ('graphsage_1/sageencoder_1/meanaggregator_2/dense_4/kernel/Adam', [256, 128]), ('graphsage_1/sageencoder_1/meanaggregator_2/dense_4/kernel/Adam_1', [256, 128]), ('graphsage_1/sageencoder_2/meanaggregator_3/dense_5/kernel', [30, 128]), ('graphsage_1/sageencoder_2/meanaggregator_3/dense_5/kernel/Adam', [30, 128]), ('graphsage_1/sageencoder_2/meanaggregator_3/dense_5/kernel/Adam_1', [30, 128]), ('graphsage_1/sageencoder_2/meanaggregator_3/dense_6/kernel', [30, 128]), ('graphsage_1/sageencoder_2/meanaggregator_3/dense_6/kernel/Adam', [30, 128]), ('graphsage_1/sageencoder_2/meanaggregator_3/dense_6/kernel/Adam_1', [30, 128]), ('graphsage_1/sageencoder_2/meanaggregator_4/dense_7/kernel', [256, 128]), ('graphsage_1/sageencoder_2/meanaggregator_4/dense_7/kernel/Adam', [256, 128]), ('graphsage_1/sageencoder_2/meanaggregator_4/dense_7/kernel/Adam_1', [256, 128]), ('graphsage_1/sageencoder_2/meanaggregator_4/dense_8/kernel', [256, 128]), ('graphsage_1/sageencoder_2/meanaggregator_4/dense_8/kernel/Adam', [256, 128]), ('graphsage_1/sageencoder_2/meanaggregator_4/dense_8/kernel/Adam_1', [256, 128])]

这样子也找不到呀，我需要人工指定feature embedidng的名字？然后在找到么？

alinamimi commented 5 years ago

sparse_feature_max_id=[29,321],

sparse_feature_idx, sparse_feature_max_id,use_hash_embedding 三个都需要给

wentixiaogege commented 5 years ago

sparse_feature_idx表示了？ buf["uint64_feature"][0] = list(feature[feature.id == node].price_cat_lable.values) buf["uint64_feature"][1] = list(feature[feature.id == node].cate3_label.values)

这里的[0,1] 么？

wentixiaogege commented 5 years ago 
layers.HashSparseEmbedding if use_hash else layers.SparseEmbedding

AttributeError: 'module' object has no attribute 'HashSparseEmbedding'

wentixiaogege commented 5 years ago

use_hash_embedding 这个要设置成False么？

zonghua94 commented 5 years ago

use_hash_embedding 这个要设置成False么？

use_hash_embedding不需要设置， 需要设置sparse_feature_idx, sparse_feature_max_id这两个参数的值

wentixiaogege commented 5 years ago

sparse_feature_idx ？ buf["uint64_feature"][0] = list(feature[feature.id == node].price_cat_lable.values) buf["uint64_feature"][1] = list(feature[feature.id == node].cate3_label.values)

这里的[0,1] 么？

zonghua94 commented 5 years ago

sparse_feature_idx ？ buf["uint64_feature"][0] = list(feature[feature.id == node].price_cat_lable.values) buf["uint64_feature"][1] = list(feature[feature.id == node].cate3_label.values)

这里的[0,1] 么？

是的

wentixiaogege commented 5 years ago

我正在跑，可是我感觉 tf https://blog.csdn.net/wuguangbin1230/article/details/79101174/ 这种link都读取不到咱们这个embedding的数据呢。。。你有demo么****

wentixiaogege commented 5 years ago

SparseSageEncoder 是否我要切换成这个encoder呢？

wentixiaogege commented 5 years ago
更新一下进展

step1##################################修改models.py文件 class GraphSage(base.UnsupervisedModel): def init(self, node_type, edge_type, max_id, metapath, fanouts, dim, aggregator='mean', concat=False, feature_idx=-1, feature_dim=0, use_feature=None, use_id=False, sparse_feature_idx=-1, sparse_feature_max_id=-1, embedding_dim=16, use_hash_embedding=False, use_residual=False, *args, *kwargs): super(GraphSage, self).init( node_type, edge_type, max_id, args, **kwargs) self._target_encoder = encoders.SparseSageEncoder( metapath=metapath, feature_ixs=sparse_feature_idx,feature_dims=sparse_feature_max_id,fanouts=fanouts, dim=dim, aggregator=aggregator, concat=concat, feature_idx=feature_idx, feature_dim=feature_dim, max_id=max_id, use_id=use_id, sparse_feature_idx=sparse_feature_idx, sparse_feature_max_id=sparse_feature_max_id, embedding_dim=embedding_dim, use_hash_embedding=use_hash_embedding, use_residual=use_residual) self._context_encoder = encoders.SparseSageEncoder( metapath=metapath, feature_ixs=sparse_feature_idx,feature_dims=sparse_feature_max_id, fanouts=fanouts, dim=dim, aggregator=aggregator, concat=concat, feature_idx=feature_idx, feature_dim=feature_dim, max_id=max_id, use_id=use_id, sparse_feature_idx=sparse_feature_idx, sparse_feature_max_id=sparse_feature_max_id, embedding_dim=embedding_dim, use_hash_embedding=use_hash_embedding, use_residual=use_residual) step2##################################################修改run_loop.py文件 elif flags_obj.model == 'graphsage': model = models.GraphSage( node_type=flags_obj.train_node_type, edge_type=flags_obj.train_edge_type, max_id=flags_obj.max_id, xent_loss=flags_obj.xent_loss, num_negs=flags_obj.num_negs, metapath=metapath, fanouts=fanouts, dim=flags_obj.dim, aggregator=flags_obj.aggregator, concat=flags_obj.concat, feature_idx=flags_obj.feature_idx, feature_dim=flags_obj.feature_dim, sparse_feature_max_id=[29,321], sparse_feature_idx=[0,1]) step3################################################# 执行训练 step4###############################################只能读取检查点的 import tensorflow as tf from tensorflow.python import pywrap_tensorflow

首先，使用tensorflow自带的python打包库读取模型

model_reader = pywrap_tensorflow.NewCheckpointReader(r"./ckpt/model.ckpt-660")

然后，使reader变换成类似于dict形式的数据

var_dict = model_reader.get_variable_to_shape_map()

最后，循环打印输出

checking_dict={} for key in var_dict: print("variable name: ", key) if 'sparseembedding' in key: checking_dict[key]=model_reader.get_tensor(key) print(model_reader.get_tensor(key))

step5################################################得到一个embedding信息 ('variable name: ', 'graphsage_1/sparsesageencoder_1/sparseembedding_1/embeddings/Adam') [[ 4.64162640e-02 -8.10544007e-04 3.02108675e-02 -1.95793472e-02 4.51767668e-02 -1.45201281e-01 1.96660638e-01 4.76137474e-02 -3.38764675e-03 7.69856572e-02 2.96429265e-03 -8.03795736e-03 1.37849394e-02 7.40052015e-02 2.60055996e-04 -2.06987374e-03] [ 1.34500396e-02 1.01608979e-02 2.89536528e-02 1.68182366e-02 3.10262516e-02 -7.32485056e-02 8.48646984e-02 5.18619232e-02 -9.40037612e-03 4.24668044e-02 1.86010413e-02 1.52076967e-02 1.59960985e-02 4.11679931e-02 7.59923132e-03 -1.03252549e-02]

这样就可以得到这个embedding信息了；但是现在有这么多；到底那个才是我要的啊；；

求指导~

['graphsage_1/sparsesageencoder_2/sparseembedding_4/embeddings', 'graphsage_1/sparsesageencoder_1/sparseembedding_1/embeddings/Adam', 'graphsage_1/sparsesageencoder_2/sparseembedding_3/embeddings/Adam_1', 'graphsage_1/sparsesageencoder_1/sparseembedding_1/embeddings', 'graphsage_1/sparsesageencoder_1/sparseembedding_2/embeddings/Adam', 'graphsage_1/sparsesageencoder_2/sparseembedding_3/embeddings/Adam', 'graphsage_1/sparsesageencoder_2/sparseembedding_4/embeddings/Adam', 'graphsage_1/sparsesageencoder_2/sparseembedding_4/embeddings/Adam_1', 'graphsage_1/sparsesageencoder_1/sparseembedding_1/embeddings/Adam_1', 'graphsage_1/sparsesageencoder_1/sparseembedding_2/embeddings/Adam_1', 'graphsage_1/sparsesageencoder_1/sparseembedding_2/embeddings', 'graphsage_1/sparsesageencoder_2/sparseembedding_3/embeddings']

wentixiaogege commented 5 years ago

您好，我没理解错的话，是表粗的这两个对么 ('key is ', 'graphsage_1/sparsesageencoder_2/sparseembedding_4/embeddings') ('value shape is', (323, 16)) ('key is ', 'graphsage_1/sparsesageencoder_1/sparseembedding_1/embeddings') ('value shape is', (31, 16)) ('key is ', 'graphsage_1/sparsesageencoder_1/sparseembedding_2/embeddings') ('value shape is', (323, 16)) ('key is ', 'graphsage_1/sparsesageencoder_2/sparseembedding_3/embeddings') ('value shape is', (31, 16))

wentixiaogege commented 5 years ago

好像这个类的super方法也得改一下，加上**kwargs

class SparseSageEncoder(SageEncoder): """ """

@staticmethod def create_sparse_embeddings(feature_dims): sparse_embeddings = [ layers.SparseEmbedding(feature_dim + 1, 16) for feature_dim in feature_dims ] return sparse_embeddings

def init(self, metapath, fanouts, dim, feature_ixs, feature_dims, shared_embeddings=None, aggregator='mean', concat=False, shared_aggregators=None, kwargs): super(SparseSageEncoder, self).init( metapath, fanouts, dim, shared_aggregators=shared_aggregators, use_feature=False, kwargs) self.feature_ixs = feature_ixs self.feature_dims = feature_dims self.dims[0] = 16 * len(feature_ixs)

wentixiaogege commented 5 years ago

@alinamimi @zonghua94 @formath @zakheav

wentixiaogege commented 5 years ago

经过确认上面的流程是对的，close this issue

aimetrics commented 4 years ago

经过确认上面的流程是对的，close this issue

@wentixiaogege 请问你有看到layers.HashEmbedding和layers.HashSparseEmbedding的源代码在哪里吗?